Module Expr Str Import alias Import alias Import alias Import alias Import alias Import alias Import alias TryExcept body Import alias handlers ExceptHandler type NameLoad body Assign NameStore NameLoad orelse ImportFrom alias ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore ListLoad Str Str ClassDef bases AttributeLoad NameLoad attr body Pass decorator_list FunctionDef arguments args NameParam defaults body For NameStore NameLoad body TryExcept body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad handlers ExceptHandler type NameLoad body Pass Raise Call NameLoad decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad NameLoad body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad If CompareEq TupleLoad NameLoad NameLoad TupleLoad NameLoad NameLoad body Raise Call NameLoad Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr BoolOpOr NameLoad Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body If AttributeLoad NameLoad attr body Return AttributeLoad NameLoad attr orelse Return Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body If AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str orelse Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Str If AttributeLoad NameLoad attr body Assign NameStore Str Assign NameStore Str If AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad Str attr AttributeLoad NameLoad attr Return Call AttributeLoad Str attr keyword AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Return Call AttributeLoad Str attr keyword NameLoad keyword AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad If CompareIs NameLoad NameLoad body Raise Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore NameStore NameLoad If CompareEq Call NameLoad SubscriptLoad NameLoad Index Str Call NameLoad Str body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad NameLoad Index Str Str orelse Assign NameStore Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad NameLoad Index Str Str attr Return Call NameLoad SubscriptLoad NameLoad Index Str keyword Call AttributeLoad NameLoad attr Str keyword NameLoad keyword NameLoad keyword Call NameLoad SubscriptLoad NameLoad Index Str decorator_list NameLoad decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad keyword AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str FunctionDef arguments args defaults body If AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr orelse Assign NameStore NameLoad For NameStore Call AttributeLoad NameLoad attr NameLoad body If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad orelse Assign NameStore NameLoad If CompareIsNot NameLoad NameLoad body Expr Yield NameLoad decorator_list Return Call NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list NameLoad decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str FunctionDef arguments args defaults body If AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr orelse Assign NameStore NameLoad With Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Str NameStore body For NameStore NameLoad body If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad orelse Assign NameStore NameLoad If CompareIsNot NameLoad NameLoad body Expr Yield NameLoad decorator_list Return Call NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr keyword NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr decorator_list decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore ListLoad For TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad body For NameStore NameLoad body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Str NameLoad Continue Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore ListLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad For NameStore NameLoad body Expr Call AttributeLoad SubscriptLoad NameLoad Index AttributeLoad NameLoad attr attr NameLoad For NameStore Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr keyword Lambda arguments args NameParam defaults AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AugAssignAdd NameStore NameLoad Assign NameStore Call AttributeLoad BinOpSub Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr keyword Num attr If CompareLt AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore ListLoad For NameStore Call NameLoad NameLoad body If AttributeLoad NameLoad attr body Continue Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore ListLoad For NameStore Call NameLoad NameLoad body If BoolOpAnd NameLoad UnaryOpNot AttributeLoad NameLoad attr body Continue Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Str AttributeLoad NameLoad attr With Call AttributeLoad NameLoad attr NameLoad Str NameStore body For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Str Return Call AttributeLoad Str attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore NameLoad While CompareLtE NameLoad NameLoad body Expr Yield NameLoad AugAssignAdd NameStore Call AttributeLoad NameLoad attr keyword Num decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign NameStore ListLoad For NameStore Call NameLoad NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad NameLoad Assign NameStore ListLoad For NameStore NameLoad body For NameStore Call AttributeLoad NameLoad attr NameLoad body TryExcept body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr Str NameLoad NameLoad Continue Assign NameStore ListLoad For NameStore NameLoad body Assert CompareEq AttributeLoad NameLoad attr NameLoad If CompareIs AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad orelse If BoolOpAnd CompareGtE AttributeLoad NameLoad attr NameLoad CompareLtE AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign NameStore Call NameLoad NameLoad Assign NameStore ListLoad For NameStore NameLoad body If CompareIs AttributeLoad NameLoad attr NameLoad body If BoolOpOr CompareLt AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr CompareGt AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr body Continue orelse If BoolOpOr CompareLt AttributeLoad NameLoad attr NameLoad CompareGt AttributeLoad NameLoad attr NameLoad body Continue Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword Call NameLoad Return Call AttributeLoad NameLoad attr NameLoad decorator_list
N/A N/A 
blueox.store
~~~~~~~~

This module provides utility functions for managing log files as created by
oxd.

:copyright: (c) 2015 by Rhett Garber
:license: ISC, see LICENSE for more details.

 N/A logging N/A datetime N/A os N/A re N/A collections N/A io N/A bz2 N/A N/A N/A boto N/A N/A N/A ImportError N/A N/A boto None N/A boto.s3.connection OrdinaryCallingFormat N/A errors N/A log N/A N/A logging getLogger __name__ N/A DATE_FORMATS N/A %Y%m%d %Y%m%d %H:%M InvalidDateError N/A N/A errors Error N/A N/A N/A parse_date_range_argument N/A N/A value N/A N/A N/A fmt DATE_FORMATS N/A N/A N/A N/A N/A N/A N/A datetime datetime strptime value fmt N/A N/A N/A ValueError N/A N/A N/A N/A InvalidDateError N/A LogFile N/A object N/A N/A Represents a log file

    Primarily this is useful for generation and interpretation of the bits of
    data that form the log's filename and path. Sub-classes implement
    functionality to actually manipulate log files.
     __init__ N/A N/A self type_name host dt date bzip N/A None None None False N/A N/A N/A self type_name type_name N/A N/A self host host N/A N/A self bzip bzip N/A N/A N/A dt date N/A None None N/A N/A N/A ValueError Needs a date N/A N/A self dt dt N/A N/A self date N/A date N/A N/A dt date N/A sort_dt N/A N/A self N/A N/A N/A N/A self dt N/A N/A N/A self dt N/A N/A N/A N/A datetime datetime N/A N/A self date year N/A N/A self date month N/A N/A self date day N/A property file_name N/A N/A self N/A N/A N/A N/A self dt N/A N/A date_name_str N/A N/A N/A self dt strftime %Y%m%d%H N/A N/A date_name_str N/A N/A N/A self date strftime %Y%m%d N/A bzip_str N/A N/A self bzip N/A N/A bzip_str .bz2 N/A host_str N/A N/A self host N/A N/A host_str N/A N/A -{} format N/A self host N/A N/A N/A {type}-{date_name}{host_str}.log{bzip} format type N/A self type_name host_str host_str date_name date_name_str bzip bzip_str N/A property file_path N/A N/A self N/A N/A N/A date_str N/A N/A N/A self date strftime %Y%m%d N/A N/A N/A {date}/{filename} format date date_str filename N/A self file_name N/A property from_filename N/A N/A cls filename N/A N/A N/A basename N/A N/A N/A os path basename filename N/A match N/A N/A re match ^(?P<stream>.+)\-(?P<date>\d{8,10})\-?(?P<host>.+)?\.log(?P<zip>\.bz2|\.gz)?$ basename N/A N/A match None N/A N/A N/A ValueError basename N/A match_info N/A N/A match groupdict N/A log_dt log_date None N/A N/A N/A len N/A match_info N/A date N/A len yyyymmddHH N/A N/A log_dt N/A N/A N/A datetime datetime strptime N/A match_info N/A date %Y%m%d%H N/A N/A log_date N/A N/A N/A N/A N/A datetime datetime strptime N/A match_info N/A date %Y%m%d date N/A N/A cls N/A match_info N/A stream host N/A N/A match_info get host dt log_dt date log_date bzip N/A bool N/A match_info N/A zip N/A classmethod N/A S3LogFile N/A LogFile N/A s3_key N/A N/A self bucket N/A N/A N/A N/A N/A N/A N/A boto s3 key Key bucket name N/A self file_path N/A open N/A N/A self bucket N/A N/A N/A Create a iterable stream of data from the log file.

        Automatically handles bzip decoding
         stream N/A N/A N/A N/A N/A N/A self bzip N/A N/A decompressor N/A N/A bz2 BZ2Decompressor N/A N/A decompressor None N/A data N/A N/A self s3_key bucket N/A N/A decompressor N/A N/A r N/A N/A decompressor decompress data N/A N/A r data N/A N/A r None N/A N/A N/A r N/A N/A N/A stream N/A from_s3_key N/A N/A cls key N/A N/A N/A N/A N/A cls from_filename N/A key name N/A classmethod N/A LocalLogFile N/A LogFile N/A get_local_file_path N/A N/A self log_path N/A N/A N/A N/A N/A N/A os path join log_path N/A self file_path N/A open N/A N/A self log_path N/A N/A N/A Create a iterable stream of data from the log file.

        Automatically handles bzip decoding
         stream N/A N/A N/A N/A N/A N/A self bzip N/A N/A decompressor N/A N/A bz2 BZ2Decompressor N/A N/A decompressor None N/A N/A N/A io open N/A N/A self get_local_file_path log_path rb f N/A N/A data f N/A N/A decompressor N/A N/A r N/A N/A decompressor decompress data N/A N/A r data N/A N/A r None N/A N/A N/A r N/A N/A N/A stream N/A build_remote N/A N/A self host N/A N/A N/A N/A S3LogFile N/A self type_name host host dt N/A self dt date N/A self date bzip N/A self bzip N/A N/A list_log_files N/A N/A log_path N/A N/A N/A Find and parse all the log files in the specified log path N/A log_files N/A N/A N/A dirpath dirnames filenames N/A N/A os walk log_path N/A N/A filename filenames N/A N/A full_path N/A N/A N/A os path join dirpath filename N/A N/A N/A log_file N/A N/A LocalLogFile from_filename filename N/A N/A N/A ValueError N/A N/A N/A N/A log warning Not a blueox log file: %s full_path N/A N/A N/A N/A log_files append log_file N/A log_files N/A filter_log_files_for_active N/A N/A log_files N/A N/A N/A Filter our list of log files to remove those we expect might be active. N/A out_log_files N/A N/A files_by_type N/A N/A collections defaultdict list N/A lf log_files N/A N/A N/A N/A N/A files_by_type N/A N/A lf type_name append lf N/A type_files N/A N/A files_by_type values N/A N/A N/A N/A type_files sort key N/A N/A N/A f N/A N/A f sort_dt N/A last_lf N/A N/A type_files pop N/A out_log_files type_files N/A cutoff_date N/A N/A N/A N/A N/A N/A datetime datetime utcnow N/A N/A datetime timedelta hours 1 date N/A N/A N/A last_lf date cutoff_date N/A N/A N/A N/A out_log_files append last_lf N/A out_log_files N/A filter_log_files_for_zipping N/A N/A log_files N/A N/A N/A Identify unzipped log files that are approporate for zipping.

    Each unique log type found should have the most recent log file unzipped
    as it's probably still in use.
     N/A out_files N/A N/A lf N/A filter_log_files_for_active log_files N/A N/A N/A lf bzip N/A N/A N/A N/A N/A out_files append lf N/A out_files N/A filter_log_files_for_uploading N/A N/A log_files zipped_only N/A N/A N/A Filter out log files that we shouldn't upload

    specify zipped_only if we should only bother to upload zipped log files
     N/A out_files N/A N/A lf N/A filter_log_files_for_active log_files N/A N/A N/A zipped_only N/A N/A lf bzip N/A N/A N/A N/A N/A out_files append lf N/A out_files N/A zip_log_file N/A N/A log_file log_path N/A N/A N/A orig_path N/A N/A log_file get_local_file_path log_path N/A N/A log_file bzip True N/A zip_path N/A N/A log_file get_local_file_path log_path N/A zip_file N/A N/A bz2 BZ2File zip_path w N/A io DEFAULT_BUFFER_SIZE N/A N/A N/A io open orig_path rb fp N/A N/A data fp N/A N/A N/A N/A zip_file write data N/A N/A N/A zip_file close N/A N/A N/A os unlink orig_path N/A s3_prefix_for_date_and_type N/A N/A date type_name N/A N/A N/A date_str N/A N/A date strftime %Y%m%d N/A N/A N/A {}/{}- format date_str type_name N/A inclusive_date_range N/A N/A start_dt end_dt N/A N/A N/A start_date N/A N/A start_dt date N/A end_date N/A N/A end_dt date N/A current_date start_date N/A N/A current_date end_date N/A N/A N/A current_date N/A current_date N/A N/A datetime timedelta days 1 N/A find_log_files_in_s3 N/A N/A bucket type_name start_dt end_dt N/A N/A N/A prefixes N/A N/A dt N/A inclusive_date_range start_dt end_dt N/A N/A N/A N/A prefixes append N/A s3_prefix_for_date_and_type dt type_name N/A log_files N/A N/A pf prefixes N/A N/A key N/A N/A bucket list pf N/A N/A N/A N/A N/A N/A log_files append N/A N/A S3LogFile from_s3_key key N/A N/A N/A ValueError N/A e N/A N/A N/A N/A log warning S3 key %r not a log file: %r key e N/A N/A out_log_files N/A N/A lf log_files N/A N/A N/A N/A lf type_name type_name N/A N/A N/A lf dt None N/A N/A N/A N/A out_log_files append lf N/A N/A N/A N/A N/A lf dt start_dt N/A N/A lf dt end_dt N/A N/A N/A N/A out_log_files append lf N/A out_log_files N/A find_log_files_in_path N/A N/A log_path type_name start_dt end_dt N/A N/A N/A log_files N/A list_log_files log_path N/A out_log_files N/A N/A lf log_files N/A N/A N/A N/A lf dt None N/A N/A N/A N/A N/A lf date N/A N/A start_dt date N/A N/A lf date N/A N/A end_dt date N/A N/A N/A N/A N/A N/A N/A lf dt start_dt N/A N/A lf dt end_dt N/A N/A N/A N/A N/A out_log_files append lf N/A out_log_files N/A open_bucket N/A N/A bucket_name N/A N/A N/A region_name N/A N/A N/A os environ get AWS_DEFAULT_REGION us-east-1 N/A conn N/A N/A N/A boto s3 connect_to_region region_name calling_format N/A OrdinaryCallingFormat N/A N/A N/A conn get_bucket bucket_name N/A