Module Import alias Import alias Import alias ImportFrom alias identifier ClassDef bases NameLoad body Assign NameStore Str Assign NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Str Str Str Str Str Assign NameStore Dict Str Str Str ListLoad Str Str Str Str Str Str Str Str Str Str Assign NameStore Dict Str Str Str ListLoad Str Str Str Str Str Str Str Str Str Str Str Assign NameStore BinOpAdd Str Str Assign NameStore Dict Str Str Str ListLoad Str Str Str Str Str Str Str Assign NameStore Dict Str Str Str ListLoad Str Str Str Str Str Assign NameStore BinOpAdd Str Str Assign NameStore Dict Str Str Str ListLoad Str Str Str Str Str Str Str Assign NameStore Dict Str Str Str ListLoad Str Str Str Str Str Assign NameStore BinOpAdd BinOpAdd Str Str Str Assign NameStore Dict Str Str Str ListLoad Str Str Str Str Str Str Assign NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Str Str Str Str Str Str Assign NameStore Dict Str Str Str ListLoad Str Str Str Str Str Str Str Assign NameStore Dict Str Str Str ListLoad Str Str Str Str Str Str Str FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore BinOpAdd Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str Dict Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore BinOpAdd Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str Dict Str Str Str Str Str ListLoad Str Str Str ListLoad Dict Str Str Str Str Dict Str Str Str Str Dict Str Str Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore BinOpAdd Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd BinOpAdd Str Str Str AttributeLoad NameLoad attr Assign NameStore BinOpAdd AttributeLoad NameLoad attr NameLoad Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore BinOpAdd Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd BinOpAdd Str Str Str AttributeLoad NameLoad attr Assign NameStore BinOpAdd AttributeLoad NameLoad attr NameLoad Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore BinOpAdd Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Str AttributeLoad NameLoad attr Str Str Assign NameStore BinOpAdd AttributeLoad NameLoad attr NameLoad Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd Str AttributeLoad NameLoad attr Assign NameStore BinOpAdd AttributeLoad NameLoad attr NameLoad Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Num Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore BinOpAdd Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore BinOpAdd Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Str AttributeLoad NameLoad attr Str Str Assign NameStore BinOpAdd AttributeLoad NameLoad attr NameLoad Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd Str AttributeLoad NameLoad attr Assign NameStore BinOpAdd AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpAdd Str Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str Assign NameStore Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Str AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore BinOpAdd AttributeLoad NameLoad attr NameLoad Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str Dict Str Str Dict Str Str Str Str Str AttributeLoad NameLoad attr Dict Str Str Str Str Str AttributeLoad NameLoad attr Dict Str Str Str Str Str AttributeLoad NameLoad attr Dict Str Str Str Str Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Str AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore BinOpAdd AttributeLoad NameLoad attr NameLoad Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str Dict Str Str Dict Str Str Str Str Str AttributeLoad NameLoad attr Dict Str Str Str Str Str AttributeLoad NameLoad attr Dict Str Str Str Str Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Assign NameStore BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AugAssignAdd SubscriptStore NameLoad Index Str ListLoad Str Str Str Str Str Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AugAssignAdd SubscriptStore NameLoad Index Str ListLoad Str Str Str Str Assign NameStore Dict Str Str Str ListLoad Dict Str Str Str Str Str Dict Str Str Dict Str Str Str Str Str AttributeLoad NameLoad attr Dict Str Str Str Str Str NameLoad Dict Str Str Str Str Str NameLoad Dict Str Str Str Str Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body If NameLoad body Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad If NameLoad body Assign AttributeStore NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body If NameLoad body Assign AttributeStore NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Num Expr Call AttributeLoad NameLoad attr NameLoad SubscriptLoad NameLoad Index Num If NameLoad body Assign AttributeStore NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Num Expr Call AttributeLoad NameLoad attr NameLoad SubscriptLoad NameLoad Index Num decorator_list Call AttributeLoad NameLoad attr Str decorator_list If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr
N/A N/A os N/A copy N/A mock tests.unit.customizations.emr EMRBaseAWSCommandParamsTest BaseAWSCommandParamsTest TestAddSteps N/A BaseAWSCommandParamsTest N/A N/A prefix emr add-steps --cluster-id j-ABC --steps  N/A STREAMING_ARGS N/A N/A N/A N/A N/A Args=-files, s3://elasticmapreduce/samples/wordcount/wordSplitter.py, -mapper,wordSplitter.py, -reducer,aggregate, -input,s3://elasticmapreduce/samples/wordcount/input, -output,s3://mybucket/wordcount/output/2014-04-18/12-15-24 N/A STREAMING_HADOOP_SCRIPT_RUNNER_STEP N/A Jar Args /home/hadoop/contrib/streaming/hadoop-streaming.jar N/A -files s3://elasticmapreduce/samples/wordcount/wordSplitter.py -mapper wordSplitter.py -reducer aggregate -input s3://elasticmapreduce/samples/wordcount/input -output s3://mybucket/wordcount/output/2014-04-18/12-15-24 N/A STREAMING_HADOOP_COMMAND_RUNNER_STEP N/A Jar Args command-runner.jar N/A hadoop-streaming -files s3://elasticmapreduce/samples/wordcount/wordSplitter.py -mapper wordSplitter.py -reducer aggregate -input s3://elasticmapreduce/samples/wordcount/input -output s3://mybucket/wordcount/output/2014-04-18/12-15-24 N/A HIVE_BASIC_ARGS N/A Args=-f, s3://elasticmapreduce/samples/hive-ads/libs/model-build.q N/A HIVE_DEFAULT_SCRIPT_RUNNER_STEP N/A Jar Args s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar N/A s3://us-east-1.elasticmapreduce/libs/hive/hive-script --run-hive-script --hive-versions latest --args -f s3://elasticmapreduce/samples/hive-ads/libs/model-build.q N/A HIVE_DEFAULT_COMMAND_RUNNER_STEP N/A Jar Args command-runner.jar N/A hive-script --run-hive-script --args -f s3://elasticmapreduce/samples/hive-ads/libs/model-build.q N/A PIG_BASIC_ARGS N/A Args=-f, s3://elasticmapreduce/samples/pig-apache/do-reports2.pig N/A PIG_DEFAULT_SCRIPT_RUNNER_STEP N/A Jar Args s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar N/A s3://us-east-1.elasticmapreduce/libs/pig/pig-script --run-pig-script --pig-versions latest --args -f s3://elasticmapreduce/samples/pig-apache/do-reports2.pig N/A PIG_DEFAULT_COMMAND_RUNNER_STEP N/A Jar Args command-runner.jar N/A pig-script --run-pig-script --args -f s3://elasticmapreduce/samples/pig-apache/do-reports2.pig N/A IMPALA_BASIC_ARGS N/A N/A Args= --impala-script,s3://myimpala/input, --console-output-path,s3://myimpala/output N/A IMPALA_BASIC_SCRIPT_RUNNER_STEP N/A Jar Args s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar N/A s3://us-east-1.elasticmapreduce/libs/impala/setup-impala --run-impala-script --impala-script s3://myimpala/input --console-output-path s3://myimpala/output N/A SPARK_SUBMIT_BASIC_ARGS N/A N/A N/A N/A N/A N/A Args= [--deploy-mode, cluster, --conf, k1=v1, s3://mybucket/myfolder/app.jar, k2=v2] N/A SPARK_SUBMIT_SCRIPT_RUNNER_STEP N/A Jar Args s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar N/A /home/hadoop/spark/bin/spark-submit --deploy-mode cluster --conf k1=v1 s3://mybucket/myfolder/app.jar k2=v2 N/A SPARK_SUBMIT_COMMAND_RUNNER_STEP N/A Jar Args command-runner.jar N/A spark-submit --deploy-mode cluster --conf k1=v1 s3://mybucket/myfolder/app.jar k2=v2 test_unknown_step_type N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Type=unknown N/A expected_error_msg N/A 
aws: error:  The step type unknown is not supported.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A test_default_step_type_name_action_on_failure N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Jar=s3://mybucket/mytest.jar N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Custom JAR CONTINUE N/A Jar s3://mybucket/mytest.jar N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release expected_result N/A test_custom_jar_step_missing_jar N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Name=CustomJarMissingJar N/A expected_error_msg N/A 
aws: error: The following  required parameters are missing for CustomJARStepConfig: Jar.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A test_custom_jar_step_with_all_fields N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Name=Custom,Type=Custom_jar,Jar=s3://mybucket/mytest.jar,Args=arg1,arg2,MainClass=mymainclass,ActionOnFailure=TERMINATE_CLUSTER,Properties=k1=v1\,k2=v2\,k3 N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Custom TERMINATE_CLUSTER N/A Jar Args MainClass Properties s3://mybucket/mytest.jar N/A arg1 arg2 mymainclass N/A N/A Key Value k1 v1 N/A Key Value k2 v2 N/A Key Value k3 N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release expected_result N/A test_streaming_step_with_default_fields N/A N/A self N/A N/A N/A cmd N/A N/A N/A self prefix Type=Streaming, N/A self STREAMING_ARGS N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Streaming program CONTINUE N/A self STREAMING_HADOOP_SCRIPT_RUNNER_STEP N/A expected_result_release N/A N/A copy deepcopy expected_result N/A N/A N/A N/A expected_result_release N/A Steps N/A 0 N/A HadoopJarStep N/A self STREAMING_HADOOP_COMMAND_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release expected_result_release N/A test_streaming_step_missing_args N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Type=Streaming N/A expected_error_msg N/A 
aws: error: The following  required parameters are missing for StreamingStepConfig: Args.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A test_streaming_jar_with_all_fields N/A N/A self N/A N/A N/A test_step_config N/A N/A N/A Type=Streaming, Name=StreamingStepAllFields, ActionOnFailure=CANCEL_AND_WAIT, N/A self STREAMING_ARGS N/A cmd N/A N/A self prefix test_step_config N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep StreamingStepAllFields CANCEL_AND_WAIT N/A self STREAMING_HADOOP_SCRIPT_RUNNER_STEP N/A expected_result_release N/A N/A copy deepcopy expected_result N/A N/A N/A N/A expected_result_release N/A Steps N/A 0 N/A HadoopJarStep N/A self STREAMING_HADOOP_COMMAND_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release expected_result_release N/A test_hive_step_with_default_fields N/A N/A self N/A N/A N/A cmd N/A N/A N/A self prefix Type=Hive, N/A self HIVE_BASIC_ARGS N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Hive program CONTINUE N/A self HIVE_DEFAULT_SCRIPT_RUNNER_STEP N/A expected_result_release N/A N/A copy deepcopy expected_result N/A N/A N/A N/A expected_result_release N/A Steps N/A 0 N/A HadoopJarStep N/A self HIVE_DEFAULT_COMMAND_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release expected_result_release N/A test_hive_step_missing_args N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Type=Hive N/A expected_error_msg N/A 
aws: error: The following  required parameters are missing for HiveStepConfig: Args.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A test_hive_step_with_all_fields N/A N/A self N/A N/A N/A test_step_config N/A N/A N/A Type=Hive, ActionOnFailure=CANCEL_AND_WAIT, Name=HiveWithAllFields, N/A self HIVE_BASIC_ARGS N/A cmd N/A N/A self prefix test_step_config N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep HiveWithAllFields CANCEL_AND_WAIT N/A self HIVE_DEFAULT_SCRIPT_RUNNER_STEP N/A expected_result_release N/A N/A copy deepcopy expected_result N/A N/A N/A N/A expected_result_release N/A Steps N/A 0 N/A HadoopJarStep N/A self HIVE_DEFAULT_COMMAND_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release expected_result_release N/A test_pig_step_with_default_fields N/A N/A self N/A N/A N/A cmd N/A N/A N/A self prefix Type=Pig, N/A self PIG_BASIC_ARGS N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Pig program CONTINUE N/A self PIG_DEFAULT_SCRIPT_RUNNER_STEP N/A expected_result_release N/A N/A copy deepcopy expected_result N/A N/A N/A N/A expected_result_release N/A Steps N/A 0 N/A HadoopJarStep N/A self PIG_DEFAULT_COMMAND_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release expected_result_release N/A test_pig_missing_args N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Type=Pig N/A expected_error_msg N/A 
aws: error: The following  required parameters are missing for PigStepConfig: Args.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A test_pig_step_with_all_fields N/A N/A self N/A N/A N/A test_step_config N/A N/A N/A N/A Name=PigWithAllFields, Type=Pig, N/A self PIG_BASIC_ARGS , ActionOnFailure=CANCEL_AND_WAIT N/A cmd N/A N/A self prefix test_step_config N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep PigWithAllFields CANCEL_AND_WAIT N/A self PIG_DEFAULT_SCRIPT_RUNNER_STEP N/A expected_result_release N/A N/A copy deepcopy expected_result N/A N/A N/A N/A expected_result_release N/A Steps N/A 0 N/A HadoopJarStep N/A self PIG_DEFAULT_COMMAND_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release expected_result_release N/A test_impala_step_with_default_fields N/A N/A self N/A N/A N/A test_step_config N/A Type=Impala, N/A self IMPALA_BASIC_ARGS N/A cmd N/A N/A self prefix test_step_config N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Impala program CONTINUE N/A self IMPALA_BASIC_SCRIPT_RUNNER_STEP N/A N/A N/A self assert_params_for_cmd cmd expected_result N/A test_SPARK_SUBMIT_SCRIPT_RUNNER_STEP N/A N/A self N/A N/A N/A cmd N/A N/A N/A self prefix Type=SPARK, N/A self SPARK_SUBMIT_BASIC_ARGS N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Spark application CONTINUE N/A self SPARK_SUBMIT_SCRIPT_RUNNER_STEP N/A expected_result_release N/A N/A copy deepcopy expected_result N/A N/A N/A N/A expected_result_release N/A Steps N/A 0 N/A HadoopJarStep N/A self SPARK_SUBMIT_COMMAND_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release expected_result_release N/A test_spark_missing_arg N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Type=SPARK N/A expected_error_msg N/A 
aws: error: The following  required parameters are missing for SparkStepConfig: Args.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A test_impala_missing_args N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Type=Impala N/A expected_error_msg N/A 
aws: error: The following  required parameters are missing for ImpalaStepConfig: Args.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release None N/A test_impala_step_with_all_fields N/A N/A self N/A N/A N/A test_step_config N/A N/A N/A N/A Name=ImpalaWithAllFields, Type=Impala, N/A self IMPALA_BASIC_ARGS , ActionOnFailure=CANCEL_AND_WAIT N/A cmd N/A N/A self prefix test_step_config N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep ImpalaWithAllFields CANCEL_AND_WAIT N/A self IMPALA_BASIC_SCRIPT_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release None N/A test_impala_step_with_release N/A N/A self N/A N/A N/A test_step_config N/A Type=Impala, N/A self IMPALA_BASIC_ARGS N/A cmd N/A N/A self prefix test_step_config N/A expected_result_release N/A 
aws: error: The step type impala  is not supported.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg None expected_result_release expected_result_release N/A test_empty_step_args N/A N/A self N/A N/A N/A cmd N/A N/A self prefix Type=Streaming,Args= N/A expected_error_msg 
aws: error: The prameter Args cannot be an empty list.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A cmd N/A N/A self prefix Type=Pig,Args= N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A cmd N/A N/A self prefix Type=Hive,Args= N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A cmd N/A N/A self prefix Args= N/A expected_error_msg 
aws: error: The following required parameters are missing for CustomJARStepConfig: Jar.
 N/A N/A N/A self assert_error_for_ami_and_release_based_clusters cmd cmd expected_error_msg expected_error_msg expected_result_release expected_error_msg N/A test_all_step_types N/A N/A self N/A N/A N/A test_step_config N/A N/A N/A N/A N/A N/A N/A N/A Jar=s3://mybucket/mytest.jar   Type=Streaming, N/A self STREAMING_ARGS  Type=Hive, N/A self HIVE_BASIC_ARGS  Type=Pig, N/A self PIG_BASIC_ARGS  Type=Impala, N/A self IMPALA_BASIC_ARGS N/A cmd N/A N/A self prefix test_step_config N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Custom JAR CONTINUE N/A Jar s3://mybucket/mytest.jar N/A Name ActionOnFailure HadoopJarStep Streaming program CONTINUE N/A self STREAMING_HADOOP_SCRIPT_RUNNER_STEP N/A Name ActionOnFailure HadoopJarStep Hive program CONTINUE N/A self HIVE_DEFAULT_SCRIPT_RUNNER_STEP N/A Name ActionOnFailure HadoopJarStep Pig program CONTINUE N/A self PIG_DEFAULT_SCRIPT_RUNNER_STEP N/A Name ActionOnFailure HadoopJarStep Impala program CONTINUE N/A self IMPALA_BASIC_SCRIPT_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result expected_result expected_result_release None N/A test_all_step_types_release N/A N/A self N/A N/A N/A test_step_config N/A N/A N/A N/A N/A N/A Jar=s3://mybucket/mytest.jar   Type=Streaming, N/A self STREAMING_ARGS  Type=Hive, N/A self HIVE_BASIC_ARGS  Type=Pig, N/A self PIG_BASIC_ARGS N/A cmd N/A N/A self prefix test_step_config N/A expected_result_release N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Custom JAR CONTINUE N/A Jar s3://mybucket/mytest.jar N/A Name ActionOnFailure HadoopJarStep Streaming program CONTINUE N/A self STREAMING_HADOOP_COMMAND_RUNNER_STEP N/A Name ActionOnFailure HadoopJarStep Hive program CONTINUE N/A self HIVE_DEFAULT_COMMAND_RUNNER_STEP N/A Name ActionOnFailure HadoopJarStep Pig program CONTINUE N/A self PIG_DEFAULT_COMMAND_RUNNER_STEP N/A N/A N/A self assert_params_for_ami_and_release_based_clusters cmd cmd expected_result None expected_result_release expected_result_release N/A test_all_step_types_from_json N/A N/A self N/A N/A N/A data_path N/A N/A N/A os path join N/A N/A N/A os path dirname __file__ input_steps.json N/A cmd N/A N/A N/A self prefix file:// data_path N/A hive_script_runner_step N/A N/A copy deepcopy N/A self HIVE_DEFAULT_SCRIPT_RUNNER_STEP N/A N/A hive_script_runner_step N/A Args N/A -d INPUT=s3://elasticmapreduce/samples/hive-ads/tables -d OUTPUT=s3://mybucket/hive-ads/output/2014-04-18/11-07-32 -d LIBS=s3://elasticmapreduce/samples/hive-ads/libs N/A pig_script_runner_step N/A N/A copy deepcopy N/A self PIG_DEFAULT_SCRIPT_RUNNER_STEP N/A N/A pig_script_runner_step N/A Args N/A -p INPUT=s3://elasticmapreduce/samples/pig-apache/input -p OUTPUT=s3://mybucket/pig-apache/output/2014-04-21/20-09-28 N/A expected_result N/A JobFlowId Steps j-ABC N/A N/A Name ActionOnFailure HadoopJarStep Custom JAR step CANCEL_AND_WAIT N/A Jar s3://mybucket/mytest.jar N/A Name ActionOnFailure HadoopJarStep Streaming step CANCEL_AND_WAIT N/A self STREAMING_HADOOP_SCRIPT_RUNNER_STEP N/A Name ActionOnFailure HadoopJarStep Hive step TERMINATE_CLUSTER hive_script_runner_step N/A Name ActionOnFailure HadoopJarStep Pig step TERMINATE_CLUSTER pig_script_runner_step N/A Name ActionOnFailure HadoopJarStep Impala step CANCEL_AND_WAIT N/A self IMPALA_BASIC_SCRIPT_RUNNER_STEP N/A N/A N/A self assert_params_for_cmd cmd expected_result N/A assert_params_for_ami_and_release_based_clusters N/A N/A self grl_patch cmd expected_result expected_result_release N/A N/A N/A expected_result N/A N/A N/A grl_patch return_value None N/A N/A N/A self assert_params_for_cmd cmd expected_result N/A expected_result_release N/A N/A N/A grl_patch return_value emr-4.0 N/A N/A N/A self assert_params_for_cmd cmd expected_result_release N/A N/A N/A mock patch awscli.customizations.emr.emrutils.get_release_label assert_error_for_ami_and_release_based_clusters N/A N/A self grl_patch cmd expected_error_msg expected_result_release N/A N/A N/A expected_error_msg N/A N/A N/A grl_patch return_value None N/A result N/A N/A self run_cmd cmd 255 N/A N/A N/A self assertEquals expected_error_msg N/A result N/A 1 N/A expected_result_release N/A N/A N/A grl_patch return_value emr-4.0 N/A result N/A N/A self run_cmd cmd 255 N/A N/A N/A self assertEquals expected_result_release N/A result N/A 1 N/A N/A N/A mock patch awscli.customizations.emr.emrutils.get_release_label N/A N/A N/A __name__ __main__ N/A N/A N/A N/A unittest main