Module ImportFrom alias Expr Str Import alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr ClassDef bases AttributeLoad NameLoad attr body Expr Str Assign NameStore Num Assign NameStore Num Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareNotEq Call AttributeLoad NameLoad attr NameLoad body Raise Call NameLoad Str Assign NameStore Call NameLoad NameLoad If CompareNotIn AttributeLoad NameLoad attr NameLoad body Raise Call NameLoad Str Assign NameStore SubscriptLoad NameLoad Index AttributeLoad NameLoad attr If Call NameLoad NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str If CompareGt Call NameLoad NameLoad AttributeLoad NameLoad attr body Raise Call NameLoad Str If UnaryOpNot NameLoad body Raise Call NameLoad Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Assign NameStore SubscriptLoad NameLoad Index AttributeLoad NameLoad attr If Call NameLoad NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Dict For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Num Num Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore BinOpFloorDiv NameLoad Call NameLoad NameLoad If CompareEq NameLoad Num body Assign NameStore Num Assign NameStore ListLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Assign NameStore BinOpFloorDiv NameLoad NameLoad For NameStore Call NameLoad BinOpSub NameLoad Num body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Dict AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad BinOpMult NameLoad NameLoad BinOpMult NameLoad BinOpAdd NameLoad Num Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Dict AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad BinOpMult NameLoad BinOpSub NameLoad Num NameLoad Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad If UnaryOpNot AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr If CompareGt NameLoad AttributeLoad NameLoad attr body Raise Call NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr If UnaryOpNot NameLoad body Raise Call NameLoad Return TupleLoad NameLoad Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str If AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr If NameLoad body AugAssignSub NameStore Num orelse Assign NameStore AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return Dict AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return BinOpMod Str TupleLoad AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call NameLoad SubscriptLoad NameLoad Index AttributeLoad NameLoad attr SubscriptLoad NameLoad Index AttributeLoad NameLoad attr SubscriptLoad NameLoad Index AttributeLoad NameLoad attr decorator_list NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str If CompareNotIn Str AttributeLoad NameLoad attr body Assign NameStore Str If NameLoad body Raise Call AttributeLoad NameLoad attr NameLoad orelse Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call NameLoad GeneratorExp TupleLoad Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr orelse If UnaryOpNot Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad body Raise Call NameLoad Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Call NameLoad GeneratorExp TupleLoad Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr If NameLoad body Assign NameStore BinOpSub Call NameLoad Call AttributeLoad NameLoad attr NameLoad If NameLoad body Raise Call AttributeLoad NameLoad attr BinOpMod Str Call AttributeLoad Str attr NameLoad Return NameLoad decorator_list
N/A __future__ with_statement N/A 
Created on 2012/06/27

@author:  cloudysunny14@gmail.com
 N/A logging mapreduce.lib files mapreduce input_readers mapreduce errors N/A Error N/A errors Error N/A BadReaderParamsError N/A errors BadReaderParamsError GoogleStorageLineInputReader N/A N/A input_readers InputReader N/A N/A Input reader for files from a stored in the GoogleCloudStorage.

  You requires activate the google cloud storage and create bucket.
  The class shouldn't be instantiated directly. Use the split_input
  class method instead.
   N/A _MAX_SHARD_COUNT 256 N/A _MAX_FILE_PATHS_COUNT 1 N/A FILE_PATHS_PARAM file_paths N/A INITIAL_POSITION_PARAM initial_position N/A START_POSITION_PARAM start_position N/A END_POSITION_PARAM end_position N/A FILE_PATH_PARAM file_path __init__ N/A N/A self file_path start_position end_position N/A N/A N/A Initializes this instance with the given file path and character range.

    This GoogleStorageLineInputReader will read from the first record starting
    after strictly after start_position until the first record ending at or
    after end_position (exclusive). As an exception, if start_position is 0,
    then this InputReader starts reading at the first record.

    Args:
      file_path: the file_path that this input reader is processing.
      start_position: the position to start reading at.
      end_position: a position in the last record to read.
     N/A N/A self _file_path file_path N/A N/A self _start_position start_position N/A N/A self _end_position end_position N/A N/A self _has_iterated False N/A N/A self _filestream None N/A validate N/A N/A cls mapper_spec N/A N/A N/A Validates mapper spec and all mapper parameters.

    Args:
      mapper_spec: The MapperSpec for this InputReader.

    Raises:
      BadReaderParamsError: required parameters are missing or invalid.
     N/A N/A N/A N/A mapper_spec input_reader_class cls N/A N/A N/A BadReaderParamsError Mapper input reader class mismatch N/A params N/A _get_params mapper_spec N/A N/A N/A cls FILE_PATHS_PARAM params N/A N/A N/A BadReaderParamsError Must specify 'file_path' for mapper input N/A file_paths N/A params N/A N/A cls FILE_PATHS_PARAM N/A N/A isinstance file_paths basestring N/A N/A file_paths N/A N/A file_paths split , N/A N/A N/A len file_paths N/A cls _MAX_FILE_PATHS_COUNT N/A N/A N/A BadReaderParamsError Too many 'file_paths' for mapper input N/A N/A file_paths N/A N/A N/A BadReaderParamsError No 'file_paths' specified for mapper input N/A classmethod split_input N/A N/A cls mapper_spec N/A N/A N/A Returns a list of shard_count input_spec_shards for input_spec.

    Args:
      mapper_spec: The mapper specification to split from. Must contain
          'file_paths' parameter with one or more file paths.

    Returns:
      A list of GoogleStorageLineInputReader corresponding to the
      specified shards.
     N/A params N/A _get_params mapper_spec N/A file_paths N/A params N/A N/A cls FILE_PATHS_PARAM N/A N/A isinstance file_paths basestring N/A N/A file_paths N/A N/A file_paths split , N/A file_sizes N/A N/A file_path file_paths N/A N/A fp N/A N/A files BufferedFile file_path N/A N/A N/A fp seek 0 2 N/A N/A file_sizes N/A file_path N/A N/A fp tell N/A shard_count N/A min N/A cls _MAX_SHARD_COUNT N/A mapper_spec shard_count N/A shards_per_file N/A shard_count N/A len file_paths N/A N/A shards_per_file 0 N/A N/A shards_per_file 1 N/A chunks N/A N/A N/A file_path file_size N/A N/A file_sizes items N/A N/A file_chunk_size N/A file_size shards_per_file N/A i N/A xrange N/A shards_per_file 1 N/A N/A N/A N/A chunks append N/A N/A GoogleStorageLineInputReader from_json N/A N/A cls FILE_PATH_PARAM N/A cls INITIAL_POSITION_PARAM N/A cls END_POSITION_PARAM file_path N/A file_chunk_size i N/A file_chunk_size N/A i 1 N/A N/A N/A chunks append N/A N/A GoogleStorageLineInputReader from_json N/A N/A cls FILE_PATH_PARAM N/A cls INITIAL_POSITION_PARAM N/A cls END_POSITION_PARAM file_path N/A file_chunk_size N/A shards_per_file 1 file_size N/A chunks N/A classmethod next N/A N/A self N/A N/A N/A Returns the next input from as an (offset, line) tuple. N/A N/A self _has_iterated True N/A N/A N/A self _filestream N/A N/A N/A self _filestream N/A N/A files BufferedFile N/A self _file_path N/A N/A self _start_position N/A N/A N/A N/A N/A self _filestream seek N/A self _start_position N/A N/A N/A N/A self _filestream readline N/A start_position N/A N/A N/A self _filestream tell N/A N/A start_position N/A self _end_position N/A N/A N/A StopIteration N/A line N/A N/A N/A self _filestream readline N/A N/A line N/A N/A N/A StopIteration N/A N/A start_position N/A N/A line rstrip N/A _next_offset N/A N/A self N/A N/A N/A Return the offset of the next line to read. N/A N/A self _filestream N/A N/A offset N/A N/A N/A self _filestream tell N/A offset N/A N/A offset 1 N/A N/A offset N/A self _start_position N/A offset N/A to_json N/A N/A self N/A N/A N/A Returns an json-compatible input shard spec for remaining inputs. N/A N/A N/A self FILE_PATH_PARAM N/A self INITIAL_POSITION_PARAM N/A self END_POSITION_PARAM N/A self _file_path N/A N/A self _next_offset N/A self _end_position N/A __str__ N/A N/A self N/A N/A N/A Returns the string representation of this GoogleStorageLineInputReader. N/A N/A FilePath(%r):[%d, %d] N/A N/A self _file_path N/A N/A self _next_offset N/A self _end_position N/A from_json N/A N/A cls json N/A N/A N/A Instantiates an instance of this InputReader for the given shard spec. N/A N/A cls N/A json N/A N/A cls FILE_PATH_PARAM N/A json N/A N/A cls INITIAL_POSITION_PARAM N/A json N/A N/A cls END_POSITION_PARAM N/A classmethod N/A _get_params N/A N/A mapper_spec allowed_keys N/A None N/A N/A Obtain input reader parameters.

  Utility function for input readers implementation. Fetches parameters
  from mapreduce specification giving appropriate usage warnings.

  Args:
    mapper_spec: The MapperSpec for the job
    allowed_keys: set of all allowed keys in parameters as strings. If it is not
      None, then parameters are expected to be in a separate "input_reader"
      subdictionary of mapper_spec parameters.

  Returns:
    mapper parameters as dict

  Raises:
    BadReaderParamsError: if parameters are invalid/missing or not allowed.
   N/A N/A input_reader N/A mapper_spec params N/A N/A message Input reader's parameters should be specified in input_reader subdictionary. N/A allowed_keys N/A N/A N/A N/A errors BadReaderParamsError message N/A N/A N/A N/A logging warning message N/A params N/A mapper_spec params N/A params N/A dict N/A N/A N/A str n v N/A N/A n v N/A N/A params iteritems N/A N/A N/A N/A isinstance N/A N/A N/A mapper_spec params get input_reader dict N/A N/A N/A BadReaderParamsError Input reader parameters should be a dictionary N/A params N/A N/A N/A mapper_spec params get input_reader N/A params N/A dict N/A N/A N/A str n v N/A N/A n v N/A N/A params iteritems N/A allowed_keys N/A N/A params_diff N/A N/A set N/A N/A params keys allowed_keys N/A params_diff N/A N/A N/A N/A errors BadReaderParamsError N/A Invalid input_reader parameters: %s N/A N/A , join params_diff N/A params N/A