Module Expr Str Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore BinOpLShift Num Num Assign NameStore BinOpLShift Num Num Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore Str FunctionDef arguments args defaults body Return Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr keyword Call AttributeLoad NameLoad attr Str Assign NameStore ListLoad If CompareIn Str NameLoad body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str If CompareIn Str NameLoad body Assign NameStore SubscriptLoad NameLoad Index Str For NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Call AttributeLoad NameLoad attr Str body Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad NameLoad Index Str NameLoad Assign NameStore Call NameLoad NameLoad SubscriptLoad NameLoad Index Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore ListLoad If CompareIn Str NameLoad body Expr Call AttributeLoad NameLoad attr Str For TupleStore NameStore NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore NameLoad Index Str NameLoad Assign SubscriptStore NameLoad Index Str ListLoad NameLoad Assign NameStore Yield Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str If UnaryOpNot NameLoad body Assign NameStore Num Assign SubscriptStore NameLoad Index Str Num Assign NameStore Call AttributeLoad NameLoad attr Str If UnaryOpNot NameLoad body Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr NameLoad Index Str Assign SubscriptStore NameLoad Index Str NameLoad If CompareNotIn Str NameLoad body Assign SubscriptStore NameLoad Index Str AttributeLoad NameLoad attr Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad If BoolOpAnd NameLoad NameLoad body With Call AttributeLoad NameLoad attr ListComp Yield NameLoad comprehension NameStore NameLoad body Assign NameStore ListComp Yield NameLoad comprehension NameStore NameLoad orelse Assign NameStore Call NameLoad NameLoad NameLoad Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Assign NameStore Dict Str Str Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr ListLoad NameLoad Assign NameStore Yield Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Yield Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr ListLoad Str Str Expr Call AttributeLoad NameLoad attr Str keyword AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Str keyword AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr Str keyword NameLoad Expr Call AttributeLoad NameLoad attr Str keyword NameLoad Expr Call AttributeLoad NameLoad attr Str keyword AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body If CompareGt NameLoad AttributeLoad NameLoad attr body Raise Call NameLoad Str decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Assign TupleStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad If BoolOpAnd CompareEq NameLoad Num AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AttributeLoad NameLoad attr orelse Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list decorator_list
N/A N/A Pipeline stages. N/A copy mapreduce.lib.pipeline common src.clients gcs src.clients s3 src.pipelines pipeline src.pipelines shardstage src.pipelines.stages gcscompositor S3Input N/A N/A shardstage ShardStage N/A N/A Provides data from S3 as input to a pipeline. N/A CHUNK_SIZE_8MB N/A 1 23 N/A CHUNK_SIZE_32MB N/A 1 25 N/A REQUEST_CHUNK_SIZE CHUNK_SIZE_8MB N/A MAX_SHARD_SIZE CHUNK_SIZE_32MB N/A DEFAULT_CONTENT_TYPE binary/octet-stream GetHelp N/A N/A N/A N/A N/A Read files from S3.

The stage config should look like this:

```python
{
  "object": "bucket_name/object_name",
  "objects":{
    "bucket": "bucket_name",
    "prefix": "object_name_prefix",
  },
  "start": first_byte,
  "length": number_of_bytes,
  "s3Credentials": {
    "accessKey": "...",
    "accessSecret" "...",
  },
  "shardPrefix": "...",
  "sinks":[destination_object_url]
}
```

* At least one of 'object' and 'objects' must be provided.
* Within 'objects', 'bucket' is required.
* 'shardPrefix' can be used to organize the temporary objects, if any,
  created during the chunked transfer (and recomposition) of the object
  in GCS.
* Any 'sources' for this stage config will be ignored.
   N/A staticmethod run N/A N/A self config N/A N/A N/A Runs the stage.

    Args:
      config: Specifies the source object(s) and sinks.
    Yields:
      If necessary, a pipeline future.
     N/A storage N/A N/A s3 S3 config N/A N/A config get s3Credentials N/A s3_objects N/A N/A N/A object config N/A N/A N/A N/A s3_objects append N/A config N/A object N/A N/A objects config N/A N/A objects N/A config N/A objects N/A s3_obj N/A N/A storage ListBucket N/A objects N/A bucket N/A N/A objects get prefix N/A N/A N/A N/A s3_objects append N/A N/A N/A s3 S3 MakeUrl N/A objects N/A bucket s3_obj N/A s3_objects N/A zip s3_objects N/A config N/A sinks N/A N/A s3_obj gcs_obj N/A N/A s3_objects pop N/A sub_stages N/A N/A N/A objects config N/A N/A N/A N/A config pop objects N/A N/A next_s3_obj next_gcs_obj s3_objects N/A N/A cfg N/A N/A copy deepcopy config N/A N/A cfg N/A object next_s3_obj N/A N/A cfg N/A sinks N/A next_gcs_obj N/A s N/A N/A S3Input cfg N/A N/A N/A sub_stages append s N/A start N/A N/A config get start N/A N/A start N/A N/A start 0 N/A N/A config N/A start 0 N/A length N/A N/A config get length N/A N/A length N/A N/A length N/A N/A N/A storage StatObject s3_obj N/A size N/A N/A config N/A length length N/A N/A shardSize config N/A N/A N/A config N/A shardSize N/A self REQUEST_CHUNK_SIZE N/A N/A shards compositors N/A N/A self ShardStage config N/A N/A shards compositors N/A N/A N/A N/A pipeline After N/A N/A shard N/A shard shards N/A N/A _ N/A N/A compositor N/A compositor compositors N/A N/A handler N/A _S3ReadBufferHandler s3_obj gcs_obj N/A N/A config get shardPrefix N/A N/A N/A storage ReadObject url s3_obj handler N/A handler Handle start start length length N/A comp_stage N/A N/A common Ignore N/A N/A handler chunk_urls N/A N/A comp_config N/A contentType sources sinks N/A handler content_type N/A handler chunk_urls N/A gcs_obj N/A comp_stage N/A N/A N/A gcscompositor GcsCompositor comp_config N/A N/A N/A sub_stages append comp_stage N/A N/A N/A N/A common Append sub_stages N/A Lint N/A N/A self linter N/A N/A N/A Stage-specific configuration linting. N/A N/A N/A linter AtLeastOneFieldRequiredCheck N/A object objects N/A N/A N/A linter FieldCheck object validator N/A N/A s3 S3 UrlToBucketAndName N/A N/A N/A linter FieldCheck objects validator N/A N/A s3 S3 UrlToBucketAndName N/A N/A N/A linter FieldCheck s3Credentials field_type dict required True N/A N/A N/A linter FieldCheck s3Credentials.accessKey required True N/A N/A N/A linter FieldCheck s3Credentials.accessSecret required True N/A N/A N/A linter FieldCheck shardSize validator N/A self ValidateShardSize N/A ValidateShardSize N/A N/A self size N/A N/A N/A N/A size N/A self MAX_SHARD_SIZE N/A N/A N/A ValueError Size exceeds App Engine response limit. N/A N/A _S3ReadBufferHandler N/A object N/A N/A Helper for the object reader. __init__ N/A N/A self s3_obj gcs_obj shard_prefix N/A N/A N/A N/A self s3_obj s3_obj N/A N/A self gcs_obj gcs_obj N/A N/A self gcs_storage N/A N/A gcs Gcs N/A N/A gcs_bucket _ N/A N/A N/A gcs Gcs UrlToBucketAndName gcs_obj N/A N/A self url_gen N/A N/A N/A gcs Gcs UrlCreator gcs_bucket shard_prefix N/A N/A self chunk_urls N/A N/A N/A self content_type None N/A N/A self once True N/A Handle N/A N/A self buf unused_bytes_read bytes_remaining ct N/A N/A N/A The wrapped handler. N/A N/A self content_type ct N/A N/A N/A bytes_remaining 0 N/A self once N/A N/A N/A N/A N/A self gcs_storage InsertObject buf N/A self gcs_obj N/A N/A tmp N/A N/A self url_gen N/A N/A N/A N/A self chunk_urls append tmp N/A N/A N/A N/A self gcs_storage InsertObject buf tmp N/A N/A self once False N/A N/A