Module Import alias Import alias Import alias Import alias Import alias ImportFrom alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr Str Expr Str ClassDef bases AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr body Expr Str FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Str NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Str Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Str Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Assign AttributeStore NameLoad attr Call NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Return ListComp Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad comprehension NameStore Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Str Assign NameStore ListLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Str Return Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Str For NameStore Call AttributeLoad NameLoad attr NameLoad body If Call AttributeLoad NameLoad attr Str body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad Return NameLoad Raise Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str decorator_list FunctionDef arguments args NameParam defaults body Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Return Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Index Num Return Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr BinOpAdd NameLoad Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Index Num Assign NameStore NameLoad Assign NameStore NameLoad For NameStore Call AttributeLoad Call NameLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad If NameLoad body Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Index Num Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad If NameLoad body Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Index Num If CompareEq NameLoad NameLoad body Break If NameLoad body If NameLoad body Assign NameStore BinOpAdd BinOpAdd NameLoad Str NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad Return NameLoad orelse Raise Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr If NameLoad body If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr Str NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Raise Call NameLoad Str If UnaryOpNot Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Str Raise Call NameLoad Str Return NameLoad Assign NameStore Call AttributeLoad NameLoad attr If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr Str Raise Call NameLoad Str If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr Str NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Raise Call NameLoad Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad Str attr Call NameLoad NameLoad BinOpAdd BinOpAdd BinOpAdd Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad Str attr Call AttributeLoad NameLoad attr keyword NameLoad Assign NameStore ListLoad Str Str NameLoad Str Str NameLoad Str NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore ListLoad Str Str NameLoad Str NameLoad Str Expr Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body If CompareIsNot NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpAdd BinOpAdd ListLoad NameLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad Str attr Call NameLoad NameLoad NameLoad Assign NameStore BinOpAdd Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr ListLoad Str NameLoad Str NameLoad AugAssignAdd NameStore ListComp BinOpMod Str NameLoad comprehension NameStore Call AttributeLoad NameLoad attr Assign NameStore BoolOpOr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AugAssignAdd NameStore ListLoad NameLoad Str AugAssignAdd NameStore ListLoad Str Call AttributeLoad AttributeLoad NameLoad attr attr Str Str Assign TupleStore NameStore NameStore Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad AugAssignAdd NameStore NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad Str attr Call NameLoad NameLoad NameLoad Assign SubscriptStore NameLoad Index Str NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad AttributeLoad NameLoad attr keyword NameLoad For TupleStore NameStore NameStore NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list ClassDef bases AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return ListLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return Call NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return Dict decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return ListLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore ListLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr BinOpAdd Str NameLoad Expr Call AttributeLoad NameLoad attr ListComp AttributeLoad Call AttributeLoad NameLoad attr attr comprehension NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr ListLoad Str Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Return NameLoad decorator_list decorator_list
N/A N/A logging N/A os N/A re N/A subprocess N/A warnings luigi six N/A luigi.configuration N/A luigi.contrib.hadoop N/A luigi.contrib.hadoop_jar N/A luigi.contrib.hdfs luigi LocalTarget luigi.task flatten N/A logger N/A N/A logging getLogger luigi-interface N/A 
Scalding support for Luigi.

Example configuration section in luigi.cfg::

    [scalding]
    # scala home directory, which should include a lib subdir with scala jars.
    scala-home: /usr/share/scala

    # scalding home directory, which should include a lib subdir with
    # scalding-*-assembly-* jars as built from the official Twitter build script.
    scalding-home: /usr/share/scalding

    # provided dependencies, e.g. jars required for compiling but not executing
    # scalding jobs. Currently requred jars:
    # org.apache.hadoop/hadoop-core/0.20.2
    # org.slf4j/slf4j-log4j12/1.6.6
    # log4j/log4j/1.2.15
    # commons-httpclient/commons-httpclient/3.1
    # commons-cli/commons-cli/1.2
    # org.apache.zookeeper/zookeeper/3.3.4
    scalding-provided: /usr/share/scalding/provided

    # additional jars required.
    scalding-libjars: /usr/share/scalding/libjars
 ScaldingJobRunner N/A N/A N/A N/A luigi contrib hadoop JobRunner N/A N/A 
    JobRunner for `pyscald` commands. Used to run a ScaldingJobTask.
     __init__ N/A N/A self N/A N/A N/A conf N/A N/A N/A luigi configuration get_config N/A default N/A N/A N/A os environ get SCALA_HOME /usr/share/scala N/A N/A self scala_home N/A N/A conf get scalding scala-home default N/A default N/A N/A N/A os environ get SCALDING_HOME /usr/share/scalding N/A N/A self scalding_home N/A N/A conf get scalding scalding-home default N/A N/A self provided_dir N/A N/A conf get scalding scalding-provided N/A N/A N/A os path join default provided N/A N/A self libjars_dir N/A N/A conf get scalding scalding-libjars N/A N/A N/A os path join default libjars N/A N/A self tmp_dir N/A LocalTarget is_tmp True N/A _get_jars N/A N/A self path N/A N/A N/A N/A N/A N/A N/A os path join path j N/A j N/A N/A os listdir path N/A N/A j endswith .jar N/A get_scala_jars N/A N/A self include_compiler N/A False N/A N/A lib_dir N/A N/A N/A os path join N/A self scala_home lib N/A jars N/A N/A N/A N/A os path join lib_dir scala-library.jar N/A reflect N/A N/A N/A os path join lib_dir scala-reflect.jar N/A N/A N/A N/A os path exists reflect N/A N/A N/A N/A jars append reflect N/A include_compiler N/A N/A N/A N/A jars append N/A N/A N/A os path join lib_dir scala-compiler.jar N/A jars N/A get_scalding_jars N/A N/A self N/A N/A N/A lib_dir N/A N/A N/A os path join N/A self scalding_home lib N/A N/A N/A self _get_jars lib_dir N/A get_scalding_core N/A N/A self N/A N/A N/A lib_dir N/A N/A N/A os path join N/A self scalding_home lib N/A j N/A N/A os listdir lib_dir N/A N/A N/A N/A j startswith scalding-core- N/A N/A p N/A N/A N/A os path join lib_dir j N/A N/A N/A logger debug Found scalding-core: %s p N/A p N/A N/A N/A N/A N/A luigi contrib hadoop HadoopJobError Could not find scalding-core. N/A get_provided_jars N/A N/A self N/A N/A N/A N/A N/A self _get_jars N/A self provided_dir N/A get_libjars N/A N/A self N/A N/A N/A N/A N/A self _get_jars N/A self libjars_dir N/A get_tmp_job_jar N/A N/A self source N/A N/A N/A job_name N/A N/A N/A os path basename N/A N/A N/A N/A os path splitext source N/A 0 N/A N/A N/A N/A os path join N/A N/A self tmp_dir path N/A job_name .jar N/A get_build_dir N/A N/A self source N/A N/A N/A build_dir N/A N/A N/A os path join N/A N/A self tmp_dir path build N/A build_dir N/A get_job_class N/A N/A self source N/A N/A N/A job_name N/A N/A N/A N/A os path splitext N/A N/A N/A os path basename source N/A 0 N/A package None N/A job_class None N/A l N/A N/A N/A open source readlines N/A N/A p N/A N/A re search package\s+([^\s\(]+) l N/A p N/A N/A package N/A N/A N/A p groups N/A 0 N/A p N/A N/A re search class\s+([^\s\(]+).*extends\s+.*Job l N/A p N/A N/A job_class N/A N/A N/A p groups N/A 0 N/A N/A job_class job_name N/A N/A N/A job_class N/A N/A package N/A N/A job_class N/A N/A package . job_class N/A N/A N/A logger debug Found scalding job class: %s job_class N/A job_class N/A N/A N/A N/A N/A N/A luigi contrib hadoop HadoopJobError Coudl not find scalding job class. N/A build_job_jar N/A N/A self job N/A N/A N/A job_jar N/A N/A job jar N/A job_jar N/A N/A N/A N/A N/A N/A os path exists job_jar N/A N/A N/A N/A logger error Can't find jar: %s, full path %s job_jar N/A N/A N/A os path abspath job_jar N/A N/A Exception job jar does not exist N/A N/A N/A N/A job job_class N/A N/A N/A N/A logger error Undefined job_class() N/A N/A Exception Undefined job_class() N/A job_jar N/A job_src N/A N/A job source N/A N/A job_src N/A N/A N/A N/A logger error Both source() and jar() undefined N/A N/A Exception Both source() and jar() undefined N/A N/A N/A N/A N/A os path exists job_src N/A N/A N/A N/A logger error Can't find source: %s, full path %s job_src N/A N/A N/A os path abspath job_src N/A N/A Exception job source does not exist N/A job_src N/A N/A job source N/A job_jar N/A N/A self get_tmp_job_jar job_src N/A build_dir N/A N/A self get_build_dir job_src N/A N/A N/A N/A N/A os path exists build_dir N/A N/A N/A N/A os makedirs build_dir N/A classpath N/A N/A : join N/A filter None N/A N/A N/A N/A N/A self get_scalding_jars N/A N/A self get_provided_jars N/A N/A self get_libjars N/A N/A job extra_jars N/A scala_cp N/A N/A : join N/A N/A self get_scala_jars include_compiler True N/A arglist N/A java -cp scala_cp scala.tools.nsc.Main -classpath classpath -d build_dir job_src N/A N/A N/A logger info Compiling scala source: %s N/A N/A subprocess list2cmdline arglist N/A N/A N/A subprocess check_call arglist N/A arglist N/A jar cf job_jar -C build_dir . N/A N/A N/A logger info Building job jar: %s N/A N/A subprocess list2cmdline arglist N/A N/A N/A subprocess check_call arglist N/A job_jar N/A run_job N/A N/A self job tracking_url_callback N/A None N/A N/A N/A tracking_url_callback None N/A N/A N/A N/A warnings warn tracking_url_callback argument is deprecated, task.set_tracking_url is used instead. DeprecationWarning N/A job_jar N/A N/A self build_job_jar job N/A jars N/A N/A N/A job_jar N/A N/A self get_libjars N/A N/A job extra_jars N/A scalding_core N/A N/A self get_scalding_core N/A libjars N/A N/A , join N/A filter None jars N/A arglist N/A N/A N/A N/A N/A luigi contrib hdfs load_hadoop_cmd N/A jar scalding_core -libjars libjars N/A arglist N/A N/A -D%s c N/A c N/A N/A job jobconfs N/A job_class N/A N/A N/A job job_class N/A N/A self get_job_class N/A N/A job source N/A arglist N/A job_class --hdfs N/A arglist N/A --name N/A N/A N/A job task_id replace = : N/A N/A tmp_files job_args N/A N/A N/A N/A luigi contrib hadoop_jar fix_paths job N/A arglist job_args N/A env N/A N/A N/A os environ copy N/A N/A N/A jars append scalding_core N/A hadoop_cp N/A N/A : join N/A filter None jars N/A N/A env N/A HADOOP_CLASSPATH hadoop_cp N/A N/A N/A logger info Submitting Hadoop job: HADOOP_CLASSPATH=%s %s hadoop_cp N/A N/A subprocess list2cmdline arglist N/A N/A N/A N/A N/A luigi contrib hadoop run_and_track_hadoop_job arglist N/A job set_tracking_url env env N/A N/A a b tmp_files N/A N/A N/A N/A a move b N/A N/A ScaldingJobTask N/A N/A N/A N/A luigi contrib hadoop BaseHadoopJobTask N/A N/A 
    A job task for Scalding that define a scala source and (optional) main method.

    requires() should return a dictionary where the keys are Scalding argument
    names and values are sub tasks or lists of subtasks.

    For example:

    .. code-block:: python

        {'input1': A, 'input2': C} => --input1 <Aoutput> --input2 <Coutput>
        {'input1': [A, B], 'input2': [C]} => --input1 <Aoutput> <Boutput> --input2 <Coutput>
     relpath N/A N/A self current_file rel_path N/A N/A N/A 
        Compute path given current file and relative path.
         N/A script_dir N/A N/A N/A os path dirname N/A N/A N/A os path abspath current_file N/A rel_path N/A N/A N/A os path abspath N/A N/A N/A os path join script_dir rel_path N/A rel_path N/A source N/A N/A self N/A N/A N/A 
        Path to the scala source for this Scalding Job

        Either one of source() or jar() must be specified.
         N/A None N/A jar N/A N/A self N/A N/A N/A 
        Path to the jar file for this Scalding Job

        Either one of source() or jar() must be specified.
         N/A None N/A extra_jars N/A N/A self N/A N/A N/A 
        Extra jars for building and running this Scalding Job.
         N/A N/A N/A job_class N/A N/A self N/A N/A N/A 
        optional main job class for this Scalding Job.
         N/A None N/A job_runner N/A N/A self N/A N/A N/A N/A ScaldingJobRunner N/A atomic_output N/A N/A self N/A N/A N/A 
        If True, then rewrite output arguments to be temp locations and
        atomically move them into place after the job finishes.
         N/A True N/A requires N/A N/A self N/A N/A N/A N/A N/A job_args N/A N/A self N/A N/A N/A 
        Extra arguments to pass to the Scalding job.
         N/A N/A N/A args N/A N/A self N/A N/A N/A 
        Returns an array of args to pass to the job.
         N/A arglist N/A N/A N/A k v N/A N/A six iteritems N/A N/A self requires_hadoop N/A N/A N/A N/A arglist append N/A -- k N/A N/A N/A arglist extend N/A N/A N/A N/A t output path N/A t N/A flatten v N/A N/A N/A arglist extend N/A --output N/A N/A self output N/A N/A N/A arglist extend N/A N/A self job_args N/A arglist N/A N/A