Module Expr Str Assign NameStore Str Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias alias Assign NameStore Call AttributeLoad NameLoad attr keyword Num keyword Num keyword Num keyword Num keyword AttributeLoad AttributeLoad NameLoad attr attr ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Assign TupleStore NameStore AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword BinOpAdd Str Call AttributeLoad Str attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad Str Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Yield Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr keyword NameLoad If CompareIsNot NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad Call NameLoad NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr NameLoad Expr Call NameLoad decorator_list AttributeLoad NameLoad attr FunctionDef arguments args NameParam NameParam defaults body Expr Str Assert CompareIsNot AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assert CompareIsNot AttributeLoad NameLoad attr NameLoad If UnaryOpNot AttributeLoad NameLoad attr body Return For NameStore AttributeLoad NameLoad attr body If AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr ListLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign AttributeStore NameLoad attr ListLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assert CompareIsNot AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assert CompareIs AttributeLoad NameLoad attr NameLoad Str Assign NameStore Call AttributeLoad Call NameLoad AttributeLoad NameLoad attr Str attr Assign NameStore Call NameLoad Num BinOpMult BinOpDiv Call NameLoad NameLoad Num Num Expr Yield Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad Call NameLoad NameLoad AttributeLoad NameLoad attr Expr Call NameLoad decorator_list AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list decorator_list
N/A N/A Class to generate a merged log file and upload it to S3.

Can fetch pre-existing file from S3, append to it locally, and finally upload the result to S3.
Local working file is generated using tempfile.mkstemp and suffixed with the dot-joined id_list.
Name and location in S3 are based on 'id_list' and 's3_base'.
The name of the logfile in S3 will be os.path.join(s3_base, *id_list).
If multiple LocalLogMerge instances are used at the same time, id_list should be unique for each one of them.

Sample usage:
  # S3 file is: <object_store_path>/merged_log/viewfinder/full/2013-02-01/i-a5e3f
  merge = LocalLogMerge(object_store, ['2013-02-01', 'i-a5e3f'], 'merged_log/viewfinder/full/')
  yield gen.Task(merge.FetchExistingFromS3)

  for line in file:
    merge.Append(line)              # Append a line to the buffer.
  if success:
    merge.FlushBuffer()             # Apply buffer to local file.
  else:
    merge.DiscardBuffer()           # Discard buffer.
  # Process another file if needed.

  merge.Close()                     # Flush buffer and close local file.
  yield gen.Task(merge.Upload)      # Upload local file to S3
  merge.Cleanup()                   # Delete local file

 N/A __author__ marc@emailscrubbed.com (Marc Berhault) N/A logging N/A os N/A tempfile tornado gen viewfinder.backend.base retry viewfinder.backend.storage file_object_store s3_object_store N/A kS3UploadRetryPolicy N/A N/A retry RetryPolicy max_tries 5 timeout 300 min_delay 1 max_delay 30 check_exception N/A N/A retry RetryPolicy AlwaysRetryOnException LocalLogMerge N/A object N/A N/A Class used to build a single merged log file locally. __init__ N/A N/A self logs_store id_list s3_base N/A N/A N/A N/A self _logs_store logs_store N/A N/A self _s3_filename N/A N/A N/A os path join s3_base id_list N/A N/A fd N/A self _working_filename N/A N/A tempfile mkstemp suffix N/A . N/A N/A . join id_list N/A N/A self _output N/A N/A os fdopen fd w N/A N/A self _buffer N/A N/A N/A self _needs_separator False N/A FetchExistingFromS3 N/A N/A self callback N/A N/A N/A If S3 already has a file for this day/instance, fetch it and write its contents to the
    local working file.
     N/A contents N/A N/A N/A gen Task N/A N/A self _logs_store Get N/A self _s3_filename must_exist False N/A N/A contents None N/A N/A N/A N/A logging info N/A Fetched %d bytes from existing S3 merged log file %s N/A N/A len contents N/A self _s3_filename N/A N/A N/A N/A self _output write contents N/A N/A N/A N/A self _output flush N/A N/A self _needs_separator True N/A N/A callback N/A N/A gen engine Append N/A N/A self entry N/A N/A N/A Add a single entry to the buffer. N/A N/A N/A self _output None N/A N/A N/A N/A self _buffer append entry N/A FlushBuffer N/A N/A self N/A N/A N/A Write out all entries in the buffer. N/A N/A N/A self _output None N/A N/A N/A self _buffer N/A N/A N/A entry N/A self _buffer N/A N/A N/A self _needs_separator N/A N/A N/A N/A N/A self _output write N/A N/A self _needs_separator True N/A N/A N/A N/A self _output write entry N/A N/A N/A N/A self _output flush N/A N/A self _buffer N/A N/A DiscardBuffer N/A N/A self N/A N/A N/A Discard all entries in the buffer. N/A N/A self _buffer N/A N/A Close N/A N/A self N/A N/A N/A Close the working file. N/A N/A N/A self _output None N/A N/A N/A self FlushBuffer N/A N/A N/A N/A self _output close N/A N/A self _output None N/A Upload N/A N/A self callback N/A N/A N/A Upload working file to S3. N/A N/A N/A self _output None Upload called before Close. N/A contents N/A N/A N/A open N/A self _working_filename r read N/A timeout N/A max 20.0 N/A N/A N/A len contents 1024 1024 N/A N/A N/A N/A gen Task N/A retry CallWithRetryAsync kS3UploadRetryPolicy N/A N/A self _logs_store Put N/A self _s3_filename contents request_timeout timeout N/A N/A N/A logging info N/A Uploaded %d bytes to S3 file %s N/A N/A len contents N/A self _s3_filename N/A N/A callback N/A N/A gen engine Cleanup N/A N/A self N/A N/A N/A Delete the local working file. N/A N/A N/A os unlink N/A self _working_filename N/A N/A