Module Expr Str Assign NameStore Str Import alias Import alias ImportFrom alias ImportFrom alias alias Assign NameStore Num ClassDef bases NameLoad body Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad If CompareIs AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str orelse Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr If CompareIs AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Assign AttributeStore NameLoad attr Num orelse Assign AttributeStore NameLoad attr BinOpDiv BinOpMult Call NameLoad AttributeLoad NameLoad attr NameLoad AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str If CompareLt Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Assign NameStore BinOpMod Str Call AttributeLoad NameLoad attr Num AttributeLoad NameLoad attr Expr Yield TupleLoad NameLoad TupleLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str For TupleStore NameStore TupleStore NameStore NameStore Call NameLoad NameLoad body Expr Yield TupleLoad NameLoad NameLoad If CompareGtE BinOpAdd NameLoad Num AttributeLoad NameLoad attr body Break decorator_list decorator_list If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr
N/A N/A 
MapReduce job to sample n lines from a file. The mapper iterates over each
line and yields them to the reducer, combined with a random seed, so that
Hadoop will resort the lines. Then, the reducer yields the first n lines.
 N/A __author__ Benjamin Goldenberg <benjamin@yelp.com> N/A random N/A sys mrjob.job MRJob mrjob.protocol RawValueProtocol ReprProtocol N/A SAMPLING_FUDGE_FACTOR 1.2 MRLogSampler N/A MRJob N/A N/A INPUT_PROTOCOL RawValueProtocol N/A OUTPUT_PROTOCOL RawValueProtocol N/A INTERNAL_PROTOCOL ReprProtocol configure_options N/A N/A self N/A N/A N/A N/A N/A N/A super MRLogSampler self configure_options N/A N/A N/A self add_passthrough_option --sample-size type int help Number of entries to sample. N/A N/A N/A self add_passthrough_option --expected-length type int help Number of entries you expect in the log. If not specified, we'll pass every line to the reducer. N/A load_options N/A N/A self args N/A N/A N/A N/A N/A N/A super MRLogSampler self load_options args N/A N/A N/A N/A self options sample_size None N/A N/A N/A N/A N/A self option_parser error You must specify the --sample-size N/A N/A N/A self sample_size N/A N/A self options sample_size N/A N/A N/A N/A self options expected_length None N/A N/A N/A self sampling_probability 1.0 N/A N/A N/A self sampling_probability N/A N/A N/A float N/A self sample_size SAMPLING_FUDGE_FACTOR N/A N/A self options expected_length N/A mapper N/A N/A self _ line N/A N/A N/A 
        For each log line, with probability self.sampling_probability,
        yield a None key, and (random seed, line) as the value, so that
        the values get sorted randomly and fed into a single reducer.

        Args:
            line - raw log line

        Yields:
            key - None
            value - (random seed, line)
         N/A N/A N/A N/A random random N/A self sampling_probability N/A N/A seed N/A %20i N/A N/A random randint 0 N/A sys maxsize N/A N/A N/A None N/A seed line N/A reducer N/A N/A self _ values N/A N/A N/A 
        Now that the values have a random number attached,
        they'll come in in random order, so we yield the
        first n lines, and return early.

        Args:
            values - generator of (random_seed, line) pairs

        Yields:
            key - None
            value - random sample of log lines
         N/A N/A line_num N/A seed line N/A enumerate values N/A N/A N/A N/A None line N/A N/A N/A line_num 1 N/A self sample_size N/A N/A N/A N/A N/A N/A __name__ __main__ N/A N/A N/A N/A MRLogSampler run