Module Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias Import alias Assign NameStore NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Assign NameStore Num Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Dict Assign NameStore Dict Str Str Str Str Str Str ListLoad Str Str Str ListLoad Str ListLoad Str ListLoad Str ListLoad Str ListLoad Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults Num Num body Assign TupleStore AttributeStore NameLoad attr AttributeStore NameLoad attr TupleLoad NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr If UnaryOpNot NameLoad body Assign NameStore Dict FunctionDef arguments args NameParam defaults body Expr Str Return Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr NameLoad decorator_list Expr Call AttributeLoad NameLoad attr BinOpMod Str AttributeLoad NameLoad attr Assign NameStore Call NameLoad Str Expr Call AttributeLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call NameLoad Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Str TryFinally body Expr Call AttributeLoad NameLoad attr Str finalbody Expr Call AttributeLoad NameLoad attr If CompareNotIn AttributeLoad NameLoad attr NameLoad body Assign SubscriptStore NameLoad Index AttributeLoad NameLoad attr ListLoad AttributeLoad NameLoad attr Expr Call NameLoad NameLoad Call NameLoad Str Assign NameStore Dict BinOpMod Str TupleLoad AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr Str Str Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad NameLoad Call NameLoad Str Expr Call NameLoad Dict Str Str NameLoad NameLoad Call NameLoad Str Assign NameStore ListLoad Str Str Str Str Str Str Str Str Str Str Assign NameStore Dict For NameStore NameLoad body Assign SubscriptStore NameLoad Index BinOpAdd BinOpAdd Str NameLoad Str Str Expr Call NameLoad NameLoad Call NameLoad Str Assign NameStore Call NameLoad Call NameLoad Str Str TryFinally body Assign NameStore ListLoad Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str Str Str Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str Str Call NameLoad Str Str Call NameLoad Str Str Call NameLoad AttributeLoad NameLoad attr Str Call NameLoad AttributeLoad NameLoad attr Str Str BinOpMod Str AttributeLoad NameLoad attr Str BinOpMod Str AttributeLoad NameLoad attr Str Str Str Str Str Str Str Str Str Str Str BinOpMod Str Call AttributeLoad NameLoad attr Str BinOpMod Str Call AttributeLoad NameLoad attr Str BinOpMod Str Call AttributeLoad NameLoad attr Str Str Str Str Str BinOpMod Str NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd NameLoad Str NameLoad Assign NameStore Dict Assign SubscriptStore NameLoad Index Str NameLoad Assign SubscriptStore NameLoad Index Str BinOpMod Str TupleLoad AttributeLoad NameLoad attr Assign SubscriptStore NameLoad Index Str Call AttributeLoad Str attr ListLoad Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Str Str Assign SubscriptStore NameLoad Index Str Str Assign SubscriptStore NameLoad Index Str Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Assign SubscriptStore NameLoad Index Str AttributeLoad NameLoad attr Assign SubscriptStore NameLoad Index Str AttributeLoad NameLoad attr If CompareIn Str AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index Str SubscriptLoad AttributeLoad NameLoad attr Index Str If NameLoad body Assign SubscriptStore NameLoad Index Str BinOpAdd Call AttributeLoad NameLoad attr Str Str Str If NameLoad body Assign NameStore AttributeLoad NameLoad attr orelse Assign NameStore Call NameLoad Call NameLoad Str Str Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad Call NameLoad NameLoad Call NameLoad NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad keyword Call NameLoad Call NameLoad Str Str keyword NameLoad keyword NameLoad Assign NameStore Dict Assign NameStore Call AttributeLoad NameLoad attr While UnaryOpNot NameLoad body TryExcept body Expr Call AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad body Pass If BoolOpOr CompareIsNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad BoolOpAnd UnaryOpNot NameLoad CompareGt BinOpSub Call AttributeLoad NameLoad attr NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr BinOpAdd Str Call AttributeLoad Call NameLoad Call NameLoad Str attr If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr BinOpAdd Str Call AttributeLoad Call NameLoad Call NameLoad Str attr Expr Call AttributeLoad NameLoad attr Raise Call NameLoad Str finalbody Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Call NameLoad NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call NameLoad Str Assign AttributeStore NameLoad attr Call NameLoad GeneratorExp TupleLoad AttributeLoad Call AttributeLoad NameLoad attr Str attr AttributeLoad Call AttributeLoad NameLoad attr Str attr comprehension NameStore Call AttributeLoad NameLoad attr Str Assign AttributeStore NameLoad attr Call NameLoad Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr AttributeLoad NameLoad attr Expr Call NameLoad AttributeLoad NameLoad attr Call NameLoad Str ListLoad Str Str Str Str BinOpAdd BinOpAdd Str AttributeLoad NameLoad attr Str BinOpAdd BinOpAdd Str AttributeLoad NameLoad attr Str Expr Call NameLoad AttributeLoad NameLoad attr Call NameLoad Str ListLoad Str Str Str Expr Call NameLoad Dict Str BinOpMod Str AttributeLoad NameLoad attr Call NameLoad Str Expr Call NameLoad NameLoad Call NameLoad Str Assign NameStore ListLoad Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr BinOpAdd Str SubscriptLoad AttributeLoad NameLoad attr Index Str Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad Call NameLoad NameLoad Call NameLoad NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad keyword Call NameLoad Call NameLoad Str Str keyword Call NameLoad Call NameLoad Str Str keyword NameLoad While NameLoad body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr BinOpAdd Str SubscriptLoad AttributeLoad NameLoad attr Index Str handlers ExceptHandler type AttributeLoad NameLoad attr body If BoolOpOr CompareIsNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad BoolOpAnd UnaryOpNot NameLoad CompareGt BinOpSub Call AttributeLoad NameLoad attr NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr BinOpAdd Str Call AttributeLoad Call NameLoad Call NameLoad Str attr If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr BinOpAdd Str Call AttributeLoad Call NameLoad Call NameLoad Str attr Expr Call AttributeLoad NameLoad attr Raise Call NameLoad Str Break orelse Expr Call AttributeLoad NameLoad attr Num Continue Expr Call AttributeLoad NameLoad attr Break Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str If BoolOpAnd AttributeLoad NameLoad attr CompareIs Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr If BoolOpAnd AttributeLoad NameLoad attr CompareIs Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr If CompareNotEq NameLoad Str body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str If CompareIs AttributeLoad NameLoad attr NameLoad body Assign AttributeStore NameLoad attr Call NameLoad Str keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str If CompareIs AttributeLoad NameLoad attr NameLoad body Assign AttributeStore NameLoad attr Call NameLoad Str AttributeLoad NameLoad attr Return AttributeLoad NameLoad attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Return AttributeLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr attr decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Assign TupleStore NameStore NameStore Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Str Return Call NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Assign TupleStore NameStore NameStore Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Str Return Call NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults AttributeLoad NameLoad attr body Expr Str Print NameLoad Str Print NameLoad Str Print NameLoad Str Print NameLoad BinOpMod Str AttributeLoad NameLoad attr Print NameLoad Str Print NameLoad Str Print NameLoad BinOpMod Str AttributeLoad NameLoad attr decorator_list decorator_list Assign NameStore NameLoad FunctionDef arguments args NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad Assign NameStore ListLoad If NameLoad body Expr Call AttributeLoad NameLoad attr ListLoad Call AttributeLoad AttributeLoad SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index Str attr attr Str Call AttributeLoad AttributeLoad SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index Str attr attr AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index Str attr attr Str Call AttributeLoad AttributeLoad SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index Str attr attr AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call NameLoad FunctionDef arguments args defaults body If NameLoad body Expr Call NameLoad NameLoad For NameStore NameLoad body Expr Call NameLoad decorator_list Assign AttributeStore NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad NameLoad Str TryFinally body Expr Call AttributeLoad NameLoad attr Str Assign NameStore SubscriptLoad BoolOpOr BoolOpAnd NameLoad TupleLoad NameLoad TupleLoad Call AttributeLoad NameLoad attr Index Num For NameStore NameLoad body Assign NameStore SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str finalbody Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad NameLoad Str TryFinally body For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad Call AttributeLoad Str attr NameLoad finalbody Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args defaults body Expr Str Global identifier If CompareIs NameLoad NameLoad body Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return NameLoad decorator_list If CompareEq NameLoad Str body Expr Str Expr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Import alias Expr Call AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr ListLoad AttributeLoad NameLoad attr If NameLoad body Assign NameStore Call NameLoad keyword Num keyword Num Expr Call AttributeLoad NameLoad attr Print AttributeLoad NameLoad attr Print AttributeLoad NameLoad attr Print Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr ImportFrom alias Expr Call Call NameLoad Expr Call AttributeLoad NameLoad attr
N/A N/A atexit N/A subprocess N/A os N/A pwd N/A logging N/A sys N/A signal N/A shutil N/A socket N/A time N/A tempfile N/A json N/A lxml.etree N/A urllib2 desktop.lib python_util desktop.lib.test_utils clear_sys_caches restore_sys_caches hadoop.fs.hadoopfs HadoopFileSystem hadoop.job_tracker LiveJobTracker N/A hadoop.cluster N/A DEBUG_HADOOP False N/A USE_STDERR N/A N/A N/A os environ get MINI_CLUSTER_USE_STDERR False N/A CLEANUP_TMP_DIR N/A N/A N/A os environ get MINI_CLUSTER_CLEANUP True N/A MAX_CLUSTER_STARTUP_TIME 120.0 N/A CLUSTER_JT_PLUGINS org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin N/A CLUSTER_TASK_SCHEDULER org.apache.hadoop.mapred.JobQueueTaskScheduler N/A CLUSTER_QUEUE_NAMES default N/A STARTUP_CONFIGS N/A N/A TEST_USER_GROUP_MAPPING N/A test chown_test notsuperuser gamma webui hue N/A test users supergroup N/A chown_test N/A notsuperuser N/A gamma N/A webui N/A supergroup N/A LOGGER N/A N/A logging getLogger __name__ MiniHadoopCluster N/A object N/A N/A 
  Manages the invocation of a MiniHadoopClusterManager from Python.
   __init__ N/A N/A self num_datanodes num_tasktrackers N/A 1 1 N/A N/A N/A N/A self _jt N/A self _fs N/A None None N/A N/A self num_datanodes num_datanodes N/A N/A self num_tasktrackers num_tasktrackers N/A start N/A N/A self extra_configs N/A None N/A N/A 
    Start a cluster as a subprocess.
     N/A N/A self tmpdir N/A N/A tempfile mkdtemp N/A N/A extra_configs N/A N/A extra_configs N/A tmppath N/A N/A filename N/A N/A N/A Creates paths in tmpdir. N/A N/A N/A N/A os path join N/A self tmpdir filename N/A N/A N/A N/A LOGGER info N/A Using temporary directory: %s N/A self tmpdir N/A in_conf_dir N/A tmppath in-conf N/A N/A N/A os mkdir in_conf_dir N/A N/A self log_dir N/A tmppath logs N/A N/A N/A os mkdir N/A self log_dir N/A f N/A file N/A N/A N/A os path join in_conf_dir hadoop-metrics.properties w N/A N/A N/A N/A N/A f write 
dfs.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
mapred.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
jvm.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
rpc.class=org.apache.hadoop.metrics.spi.NoEmitMetricsContext
 N/A N/A N/A N/A f close N/A N/A N/A self superuser TEST_USER_GROUP_MAPPING N/A N/A N/A TEST_USER_GROUP_MAPPING N/A N/A self superuser N/A N/A self superuser N/A N/A _write_static_group_mapping TEST_USER_GROUP_MAPPING N/A tmppath ugm.properties N/A core_configs N/A N/A hadoop.proxyuser.%s.groups N/A N/A self superuser N/A hadoop.proxyuser.%s.hosts N/A N/A self superuser mapred.jobtracker.plugins users,supergroup localhost CLUSTER_JT_PLUGINS N/A N/A N/A extra_configs update STARTUP_CONFIGS N/A N/A write_config core_configs N/A tmppath in-conf/core-site.xml N/A N/A write_config N/A mapred.jobtracker.taskScheduler mapred.queue.names CLUSTER_TASK_SCHEDULER CLUSTER_QUEUE_NAMES N/A tmppath in-conf/mapred-site.xml N/A hadoop_policy_keys N/A client client.datanode datanode inter.datanode namenode inter.tracker job.submission task.umbilical refresh.policy admin.operations N/A hadoop_policy_config N/A N/A policy hadoop_policy_keys N/A N/A N/A hadoop_policy_config N/A N/A N/A security. policy .protocol.acl * N/A N/A write_config hadoop_policy_config N/A tmppath in-conf/hadoop-policy.xml N/A details_file N/A file N/A tmppath details.json w+ N/A N/A N/A args N/A N/A N/A N/A os path join N/A N/A N/A N/A hadoop conf HADOOP_MR1_HOME get bin hadoop jar N/A N/A N/A N/A hadoop conf HADOOP_TEST_JAR get minicluster -writeConfig N/A tmppath config.xml -writeDetails N/A tmppath details.json -datanodes N/A str N/A self num_datanodes -tasktrackers N/A str N/A self num_tasktrackers -useloopbackhosts -D N/A hadoop.tmp.dir=%s N/A self tmpdir -D N/A mapred.local.dir=%s/mapred/local N/A self tmpdir -D mapred.system.dir=/mapred/system -D mapred.temp.dir=/mapred/temp -D jobclient.completion.poll.interval=100 -D jobclient.progress.monitor.poll.interval=100 -D fs.checkpoint.period=1 -D N/A dfs.secondary.http.address=0.0.0.0:%d N/A N/A python_util find_unused_port -D N/A dfs.thrift.address=localhost:%d N/A N/A python_util find_unused_port -D N/A jobtracker.thrift.address=localhost:%d N/A N/A python_util find_unused_port -D jobclient.completion.poll.interval=50 -D hadoop.security.authorization=true -D N/A hadoop.policy.file=%s/hadoop-policy.xml in_conf_dir N/A N/A key value N/A N/A extra_configs iteritems N/A N/A N/A N/A args append -D N/A N/A N/A args append N/A N/A key = value N/A env N/A N/A N/A env N/A HADOOP_CONF_DIR in_conf_dir N/A N/A env N/A HADOOP_OPTS N/A -Dtest.build.data=%s N/A N/A self tmpdir N/A N/A env N/A HADOOP_CLASSPATH N/A N/A : join N/A N/A N/A N/A N/A hadoop conf HADOOP_TEST_JAR get N/A N/A N/A N/A hadoop conf HADOOP_PLUGIN_CLASSPATH get N/A N/A N/A os path join N/A N/A N/A os path dirname __file__ test_jars * N/A N/A env N/A HADOOP_HEAPSIZE 128 N/A N/A env N/A HADOOP_HOME N/A N/A N/A N/A hadoop conf HADOOP_MR1_HOME get N/A N/A env N/A HADOOP_LOG_DIR N/A self log_dir N/A N/A env N/A USER N/A self superuser N/A N/A JAVA_HOME N/A os environ N/A N/A N/A env N/A JAVA_HOME N/A N/A os environ N/A JAVA_HOME N/A DEBUG_HADOOP N/A N/A N/A env N/A HADOOP_OPTS N/A N/A N/A env get HADOOP_OPTS  -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=9999 N/A USE_STDERR N/A N/A stderr N/A sys stderr N/A N/A stderr N/A file N/A tmppath stderr w N/A N/A N/A LOGGER debug N/A Starting minicluster: %s env: %s N/A N/A repr args N/A repr env N/A N/A self clusterproc N/A N/A subprocess Popen args args stdout N/A file N/A tmppath stdout w stderr stderr env env N/A details N/A N/A start N/A N/A time time N/A N/A details N/A N/A N/A N/A N/A N/A details_file seek 0 N/A details N/A N/A json load details_file N/A N/A N/A ValueError N/A N/A N/A N/A N/A N/A N/A N/A self clusterproc poll None N/A N/A DEBUG_HADOOP N/A N/A N/A N/A time time start MAX_CLUSTER_STARTUP_TIME N/A N/A N/A N/A LOGGER debug N/A stdout: N/A N/A N/A file N/A tmppath stdout read N/A N/A USE_STDERR N/A N/A N/A N/A LOGGER debug N/A stderr: N/A N/A N/A file N/A tmppath stderr read N/A N/A N/A self stop N/A N/A Exception Cluster process quit or is taking too long to start.  Aborting. N/A N/A N/A N/A details_file close N/A N/A N/A LOGGER debug Successfully started minicluster N/A N/A k v N/A N/A details iteritems N/A N/A N/A setattr self k v N/A config N/A N/A N/A lxml etree parse N/A tmppath config.xml N/A N/A self config N/A dict N/A N/A N/A N/A N/A property find ./name text N/A N/A N/A property find ./value text N/A property N/A N/A config xpath /configuration/property N/A N/A self config_dir N/A tmppath conf N/A N/A N/A os mkdir N/A self config_dir N/A N/A N/A N/A N/A hadoop conf HADOOP_CONF_DIR set_for_testing N/A self config_dir N/A N/A write_config N/A self config N/A tmppath conf/core-site.xml N/A fs.defaultFS jobclient.completion.poll.interval dfs.namenode.checkpoint.period dfs.namenode.checkpoint.dir N/A N/A hadoop.proxyuser. N/A self superuser .groups N/A N/A hadoop.proxyuser. N/A self superuser .hosts N/A N/A write_config N/A self config N/A tmppath conf/hdfs-site.xml N/A fs.defaultFS dfs.namenode.http-address dfs.namenode.secondary.http-address N/A N/A write_config N/A mapred.job.tracker N/A localhost:%d N/A self jobtracker_port N/A tmppath conf/mapred-site.xml N/A N/A write_config hadoop_policy_config N/A tmppath conf/hadoop-policy.xml N/A args N/A N/A N/A N/A N/A hadoop conf HADOOP_BIN get --config N/A self config_dir secondarynamenode N/A N/A N/A LOGGER debug N/A Starting 2NN at:  N/A N/A self config N/A dfs.secondary.http.address N/A N/A N/A LOGGER debug N/A 2NN command: %s env: %s N/A N/A repr args N/A repr env N/A N/A self secondary_proc N/A N/A subprocess Popen args args stdout N/A file N/A tmppath stdout.2nn w stderr N/A file N/A tmppath stderr.2nn w env env N/A True N/A N/A N/A N/A response N/A N/A urllib2 urlopen N/A N/A urllib2 Request N/A http:// N/A N/A self config N/A dfs.secondary.http.address N/A N/A N/A N/A urllib2 URLError N/A N/A N/A N/A N/A N/A N/A self secondary_proc poll None N/A N/A DEBUG_HADOOP N/A N/A N/A N/A time time start MAX_CLUSTER_STARTUP_TIME N/A N/A N/A N/A LOGGER debug N/A stdout: N/A N/A N/A file N/A tmppath stdout read N/A N/A USE_STDERR N/A N/A N/A N/A LOGGER debug N/A stderr: N/A N/A N/A file N/A tmppath stderr read N/A N/A N/A self stop N/A N/A Exception 2nn process quit or is taking too long to start. Aborting. N/A N/A N/A N/A N/A time sleep 1 N/A N/A N/A N/A response close N/A N/A N/A N/A LOGGER debug Successfully started 2NN N/A stop N/A N/A self N/A N/A N/A 
    Kills the cluster ungracefully.
     N/A N/A N/A self clusterproc N/A N/A N/A N/A self clusterproc poll None N/A N/A N/A N/A os kill N/A N/A self clusterproc pid N/A signal SIGKILL N/A N/A N/A N/A self clusterproc wait N/A N/A N/A self secondary_proc N/A N/A N/A N/A self secondary_proc poll None N/A N/A N/A N/A os kill N/A N/A self secondary_proc pid N/A signal SIGKILL N/A N/A N/A N/A self secondary_proc wait N/A N/A CLEANUP_TMP_DIR false N/A N/A N/A N/A logging info Cleaning up self.tmpdir.  Use $MINI_CLUSTER_CLEANUP to avoid. N/A N/A N/A shutil rmtree N/A self tmpdir N/A fs N/A N/A self N/A N/A N/A Creates a HadoopFileSystem object configured for this cluster. N/A N/A N/A self _fs None N/A N/A N/A self _fs N/A HadoopFileSystem localhost thrift_port N/A self namenode_thrift_port hdfs_port N/A self namenode_port hadoop_bin_path N/A N/A N/A N/A hadoop conf HADOOP_BIN get N/A N/A self _fs N/A property jt N/A N/A self N/A N/A N/A Creates a LiveJobTracker object configured for this cluster. N/A N/A N/A self _jt None N/A N/A N/A self _jt N/A LiveJobTracker localhost N/A self jobtracker_thrift_port N/A N/A self _jt N/A property superuser N/A N/A self N/A N/A N/A 
    Returns the "superuser" of this cluster.  
    
    This is essentially the user that the cluster was started
    with.
     N/A N/A N/A N/A pwd getpwuid N/A N/A os getuid pw_name N/A property namenode_thrift_port N/A N/A self N/A N/A N/A 
    Return the namenode thrift port.
     N/A N/A _ port N/A N/A N/A N/A self config N/A dfs.thrift.address split : N/A N/A int port N/A property jobtracker_thrift_port N/A N/A self N/A N/A N/A 
    Return the jobtracker thrift port.
     N/A N/A _ port N/A N/A N/A N/A self config N/A jobtracker.thrift.address split : N/A N/A int port N/A property dump_ini N/A N/A self fd N/A N/A sys stdout N/A N/A 
    Dumps an ini-style configuration suitable for configuring desktop
    to talk to this cluster.
    TODO(todd) eventually this should use config framework 'writeback'
    support

    @param fd: a file-like writable object
     N/A fd [hadoop] N/A fd [[hdfs_clusters]] N/A fd [[[default]]] N/A fd N/A thrift_port=%d N/A self namenode_thrift_port N/A fd [[mapred_clusters]] N/A fd [[[default]]] N/A fd N/A thrift_port=%d N/A self jobtracker_thrift_port N/A N/A N/A _shared_cluster None shared_cluster N/A N/A conf N/A False N/A N/A 
  Use a shared cluster that is initialized on demand,
  and that is torn down at process exit.

  If conf is True, then configuration is updated to
  reference the cluster, and relevant caches are cleared.

  Returns a lambda which must be called when you are
  done with the shared cluster.
   N/A cluster N/A shared_cluster_internal N/A closers N/A N/A conf N/A N/A N/A N/A closers extend N/A N/A N/A N/A N/A N/A N/A hadoop conf HDFS_CLUSTERS N/A default NN_HOST set_for_testing localhost N/A N/A N/A N/A N/A N/A hadoop conf HDFS_CLUSTERS N/A default NN_HDFS_PORT set_for_testing N/A cluster namenode_port N/A N/A N/A N/A N/A N/A hadoop conf MR_CLUSTERS N/A default HOST set_for_testing localhost N/A N/A N/A N/A N/A N/A hadoop conf MR_CLUSTERS N/A default JT_THRIFT_PORT set_for_testing N/A N/A cluster jt thrift_port N/A old_caches N/A clear_sys_caches finish N/A N/A N/A N/A N/A conf N/A N/A N/A restore_sys_caches old_caches N/A x closers N/A N/A N/A x N/A N/A N/A cluster shutdown finish N/A cluster N/A write_config N/A N/A config path variables N/A None N/A N/A 
  Minimal utility to write Hadoop-style configuration
  from a configuration map (config), into a new file
  called path.
   N/A f N/A file path w N/A N/A N/A N/A N/A f write <?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 N/A keys N/A N/A N/A variables N/A variables N/A N/A N/A config keys N/A 0 N/A name keys N/A N/A value N/A config N/A name N/A N/A N/A f write   <property>
 N/A N/A N/A f write N/A     <name>%s</name>
 name N/A N/A N/A f write N/A     <value>%s</value>
 value N/A N/A N/A f write   </property>
 N/A N/A N/A f write </configuration>
 N/A N/A N/A N/A f close N/A _write_static_group_mapping N/A N/A user_group_mapping path N/A N/A N/A 
  Create a Java-style .properties file to contain the static user -> group
  mapping used by tests.
   N/A f N/A file path w N/A N/A N/A N/A user groups N/A N/A user_group_mapping iteritems N/A N/A N/A N/A f write N/A %s = %s
 N/A user N/A N/A , join groups N/A N/A N/A N/A f close N/A shared_cluster_internal N/A N/A N/A N/A N/A 
  Manages _shared_cluster.
   N/A _shared_cluster N/A N/A _shared_cluster None N/A N/A _shared_cluster N/A MiniHadoopCluster N/A N/A N/A _shared_cluster start N/A N/A N/A atexit register N/A _shared_cluster stop N/A _shared_cluster N/A N/A N/A __name__ __main__ N/A N/A 
  It's poor form to write tests for tests (the world-wide stack
  overflow exception), so this merely tries the code.
   N/A N/A N/A logging basicConfig level N/A logging DEBUG N/A desktop N/A N/A N/A N/A N/A desktop lib conf initialize N/A N/A hadoop conf N/A True N/A N/A cluster N/A MiniHadoopCluster num_datanodes 5 num_tasktrackers 5 N/A N/A N/A cluster start N/A N/A cluster namenode_port N/A N/A cluster jobtracker_port N/A N/A N/A N/A cluster config get dfs.thrift.address N/A N/A N/A cluster dump_ini N/A sys stdout IPython.Shell IPShellEmbed N/A N/A N/A IPShellEmbed N/A N/A N/A cluster stop