Repository: Yelp/mrjob

Issue #178: Better parsing of different kinds of errors from task-attempts
Created: 2011-07-11T23:02:26Z
I just ran a job that used the with keyword without using from __future__ import with_statement. The error was in task-attempts/.../stderr and looks like this:
mr_profile_test.py:27: Warning: 'with' will become a reserved keyword in Python 2.6
  File "mr_profile_test.py", line 27
    with open('/mnt/var/log/hadoop/profile/bloop', 'w') as f:
            ^
SyntaxError: invalid syntax
Traceback (most recent call last):
  File "wrapper.py", line 16, in <module>
    check_call(sys.argv[1:])
  File "/usr/lib/python2.5/subprocess.py", line 462, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', 'mr_profile_test.py', '--step-num=0', '--reducer', '--input-protocol', 'raw_value', '--output-protocol', 'json', '--protocol', 'json']' returned non-zero exit status 1

However, mrjob only found the irrelevant part:
Traceback (most recent call last):
  File "wrapper.py", line 16, in <module>
    check_call(sys.argv[1:])
  File "/usr/lib/python2.5/subprocess.py", line 462, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', 'mr_profile_test.py', '--step-num=0', '--reducer', '--input-protocol', 'raw_value', '--output-protocol', 'json', '--protocol', 'json']' returned non-zero exit status 1

It should probably also find the relevant part, perhaps by looking for SyntaxError and backtracking to the first non-indented line.

Duplicate issue #190: Parse bootstrap 'make' errors from task-attempts
Created: 2011-07-22T22:43:41Z
I just ran a job that had this error:
tools/csv-to-myisam.c:18:19: error: mysql.h: No such file or directory
tools/csv-to-myisam.c:92: error: 'MYSQL' has not been declared
tools/csv-to-myisam.c: In function 'void exit_on_error(int, int*, const char*)':
tools/csv-to-myisam.c:93: error: 'mysql_error' was not declared in this scope
tools/csv-to-myisam.c: At global scope:
tools/csv-to-myisam.c:97: error: 'MYSQL' was not declared in this scope
tools/csv-to-myisam.c:97: error: 'mysql' was not declared in this scope
tools/csv-to-myisam.c:97: error: expected primary-expression before 'const'
tools/csv-to-myisam.c:97: error: initializer expression list treated as compound expression
tools/csv-to-myisam.c:97: error: expected ',' or ';' before '{' token
tools/csv-to-myisam.c:117: error: 'MYSQL' was not declared in this scope
tools/csv-to-myisam.c:117: error: 'mysql' was not declared in this scope
tools/csv-to-myisam.c:117: error: expected primary-expression before 'const'
tools/csv-to-myisam.c:117: error: expected primary-expression before 'const'
tools/csv-to-myisam.c:117: error: expected primary-expression before 'const'
tools/csv-to-myisam.c:117: error: initializer expression list treated as compound expression
tools/csv-to-myisam.c:117: error: expected ',' or ';' before '{' token
make: *** [tools/csv-to-myisam] Error 1
Traceback (most recent call last):
  File "wrapper.py", line 11, in <module>
    check_call('cd yelp-src-tree.tar.gz; ln -sf $(readlink -f config/emr/level.py) config/level.py; make -f Makefile.emr', shell=True, stdout=open('/dev/null', 'w'))
  File "/usr/lib/python2.5/subprocess.py", line 462, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'cd yelp-src-tree.tar.gz; ln -sf $(readlink -f config/emr/level.py) config/level.py; make -f    Makefile.emr' returned non-zero exit status 2
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:372)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:582)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:363)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)

mrjob did not find it.
Maybe we could look for the generic subprocess.CalledProcessError and just print everything from the top of the file to there.

Closing comment (2011-07-27T22:47:58Z): This is a duplicate of #178, which is more general.