Module Expr Str ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore Call NameLoad NameLoad FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Return TupleLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Return TupleLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If UnaryOpNot Call NameLoad GeneratorExp CompareIn NameLoad NameLoad comprehension NameStore TupleLoad Str Str Str body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Return Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareIn Str NameLoad body Return Assign NameStore BoolOpOr Call AttributeLoad NameLoad attr Str Dict Assign NameStore Call AttributeLoad NameLoad attr Str If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr Str Return Assign NameStore Call AttributeLoad NameLoad attr Str Assign SubscriptStore NameLoad Index Str Call NameLoad AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str For NameStore Call NameLoad AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str SubscriptLoad NameLoad Index Str Expr Yield NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareIn Str NameLoad body Return Assign NameStore Call AttributeLoad NameLoad attr NameLoad If NameLoad body Assign SubscriptStore NameLoad Index Str NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str If BoolOpAnd CompareIn Str NameLoad BoolOpOr NameLoad UnaryOpNot Call AttributeLoad SubscriptLoad NameLoad Index Str attr Str body Return Assign NameStore BoolOpOr Call AttributeLoad NameLoad attr Str Dict Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr If NameLoad body If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr Str Return orelse If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr Str Return Assign SubscriptStore NameLoad Index Str Call NameLoad AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad body Expr Str For NameStore Call NameLoad AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str SubscriptLoad NameLoad Index Str Expr Yield NameLoad decorator_list decorator_list
N/A N/A Runner mixin for counters and probable cause of failure.

This relies on passing around a *log_interpretation* dictionary, which
is described in detail in :py:mod:`mrjob.logs`.

This mixin doesn't yet handle step logs because the EMR and Hadoop runners
handle them so differently. It's up to you to fill in the 'step' field
of your log interpretation; the mixin can't do much without it because
it needs it for the job/application ID.

Your runner should generally have one log interpretation per step,
though the mixin doesn't care how or where you store them.
 logging getLogger mrjob.compat uses_yarn mrjob.logs.counters _pick_counters mrjob.logs.errors _pick_error mrjob.logs.history _interpret_history_log mrjob.logs.history _ls_history_logs mrjob.logs.task _interpret_task_logs mrjob.logs.task _ls_task_syslogs N/A log N/A getLogger __name__ _log_parsing_task_stderr N/A N/A stderr_path N/A N/A N/A N/A N/A log info N/A   Parsing task stderr: %s stderr_path N/A LogInterpretationMixin N/A object N/A N/A Mix this in to your runner class to simplify log interpretation. _stream_history_log_dirs N/A N/A self output_dir N/A None N/A N/A Yield lists of directories (usually, URIs) to search for history
        logs in.

        Usually, you'll want to add logging messages (e.g.
        'Searching for history logs in ...'

        :param output_dir: Output directory for step (optional), to look
            for logs (e.g. on Cloudera).
         N/A N/A N/A _stream_task_log_dirs N/A N/A self application_id output_dir N/A None None N/A N/A Yield lists of directories (usually, URIs) to search for task
        logs in.

        Usually, you'll want to add logging messages (e.g.
        'Searching for task syslogs in...')

        :param application_id: YARN application ID (optional), so we can ls
            the relevant subdirectory of `userlogs/` rather than the whole
            thing
        :param output_dir: Output directory for step (optional), to look
            for logs (e.g. on Cloudera).
         N/A N/A N/A _get_step_log_interpretation N/A N/A self log_interpretation N/A N/A N/A Return interpretation of the step log. Either implement
        this, or fill ``'step'`` yourself (e.g. from Hadoop binary's
        output. N/A None N/A _pick_counters N/A N/A self log_interpretation N/A N/A N/A Pick counters from our log interpretation, interpreting
        history logs if need be. N/A counters N/A _pick_counters log_interpretation N/A N/A counters N/A N/A N/A N/A log info Attempting to fetch counters from logs... N/A N/A N/A self _interpret_step_logs log_interpretation N/A counters N/A _pick_counters log_interpretation N/A N/A counters N/A N/A N/A N/A self _interpret_history_log log_interpretation N/A counters N/A _pick_counters log_interpretation N/A counters N/A _pick_error N/A N/A self log_interpretation N/A N/A N/A Pick probable cause of failure (only call this if job fails). N/A N/A N/A all N/A N/A log_type log_interpretation N/A log_type N/A job step task N/A N/A N/A N/A log info Scanning logs for probable cause of failure... N/A N/A N/A self _interpret_step_logs log_interpretation N/A N/A N/A self _interpret_history_log log_interpretation N/A N/A N/A self _interpret_task_logs log_interpretation N/A N/A _pick_error log_interpretation N/A _interpret_history_log N/A N/A self log_interpretation N/A N/A N/A Fetch history log and add 'history' to log_interpretation. N/A N/A history log_interpretation N/A N/A N/A step_interpretation N/A N/A N/A log_interpretation get step N/A N/A job_id N/A N/A step_interpretation get job_id N/A N/A job_id N/A N/A N/A N/A log warning Can't fetch history log; missing job ID N/A N/A output_dir N/A N/A step_interpretation get output_dir N/A N/A log_interpretation N/A history N/A _interpret_history_log N/A self fs N/A N/A self _ls_history_logs job_id job_id output_dir output_dir N/A _ls_history_logs N/A N/A self job_id output_dir N/A None None N/A N/A Yield history log matches, logging a message for each one. N/A match N/A _ls_history_logs N/A self fs N/A N/A self _stream_history_log_dirs output_dir output_dir job_id job_id N/A N/A N/A N/A log info N/A   Parsing history log: %s N/A match N/A path N/A N/A match N/A _interpret_step_logs N/A N/A self log_interpretation N/A N/A N/A Add *step* to the log interpretation, if it's not already there. N/A N/A step log_interpretation N/A N/A N/A step_interpretation N/A N/A self _get_step_log_interpretation log_interpretation N/A step_interpretation N/A N/A N/A log_interpretation N/A step step_interpretation N/A _interpret_task_logs N/A N/A self log_interpretation partial N/A True N/A N/A Fetch task syslogs and stderr, and add 'task' to interpretation. N/A N/A N/A task log_interpretation N/A partial N/A N/A N/A N/A log_interpretation N/A task get partial N/A N/A N/A step_interpretation N/A N/A N/A log_interpretation get step N/A N/A application_id N/A N/A step_interpretation get application_id N/A job_id N/A N/A step_interpretation get job_id N/A output_dir N/A N/A step_interpretation get output_dir N/A yarn N/A uses_yarn N/A N/A self get_hadoop_version N/A yarn N/A N/A N/A application_id N/A N/A N/A N/A log warning Can't fetch task logs; missing application ID N/A N/A N/A N/A job_id N/A N/A N/A N/A log warning Can't fetch task logs; missing job ID N/A N/A N/A log_interpretation N/A task N/A _interpret_task_logs N/A self fs N/A N/A self _ls_task_syslogs application_id application_id job_id job_id output_dir output_dir partial partial stderr_callback _log_parsing_task_stderr N/A _ls_task_syslogs N/A N/A self application_id job_id output_dir N/A None None None N/A N/A Yield task log matches, logging a message for each one. N/A match N/A _ls_task_syslogs N/A self fs N/A N/A self _stream_task_log_dirs application_id application_id output_dir output_dir application_id application_id job_id job_id N/A N/A N/A N/A log info N/A   Parsing task syslog: %s N/A match N/A path N/A N/A match N/A N/A