Module ImportFrom alias alias alias ImportFrom alias alias alias ImportFrom alias Import alias Import alias ImportFrom alias ImportFrom alias alias ImportFrom alias Import alias identifier Import alias Import alias Import alias identifier ImportFrom alias ImportFrom alias identifier ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias With Call NameLoad NameLoad body ImportFrom alias ImportFrom alias ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam defaults vararg kwarg body Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore BoolOpOr Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr Str NameLoad Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr Str Num If BoolOpAnd UnaryOpNot NameLoad BoolOpAnd NameLoad NameLoad NameLoad body Assign NameStore Call NameLoad keyword NameLoad keyword Call NameLoad NameLoad keyword NameLoad If CompareIs NameLoad NameLoad body Raise Call NameLoad Str Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad NameLoad decorator_list decorator_list FunctionDef arguments args NameParam defaults body Return Call NameLoad BinOpMod Str AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad Dict Str NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults Num body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad With Call NameLoad AttributeLoad NameLoad attr NameStore body With Call NameLoad NameLoad Str NameStore body Expr Call AttributeLoad NameLoad attr NameLoad Expr Yield NameLoad decorator_list Call AttributeLoad NameLoad attr NameLoad NameLoad FunctionDef arguments args NameParam defaults kwarg body With Call NameLoad NameLoad NameStore body Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad Return NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad FunctionDef arguments args NameParam NameParam defaults Num kwarg body With Call NameLoad NameLoad keyword NameLoad NameStore body Assign NameStore Call NameLoad Call NameLoad NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Return NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad NameLoad FunctionDef arguments args NameParam defaults kwarg body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore BinOpAdd BinOpAdd AttributeLoad NameLoad attr Str SubscriptLoad SubscriptLoad SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Index Num Index Str Assign NameStore Call Call NameLoad NameLoad NameLoad keyword AttributeLoad NameLoad attr With Call NameLoad NameLoad NameLoad NameStore body Expr Yield NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad Call NameLoad NameLoad NameLoad FunctionDef arguments args NameParam NameParam defaults Num kwarg body With Call NameLoad NameLoad keyword NameLoad NameStore body Assign NameStore Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Return NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad Call NameLoad NameLoad Expr Str Assign NameStore Call NameLoad Str Str Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam defaults Str NameLoad NameLoad kwarg body If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad With Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr body Return Call AttributeLoad NameLoad attr NameLoad NameLoad keyword NameLoad keyword NameLoad If BoolOpAnd NameLoad BoolOpOr UnaryOpNot NameLoad BoolOpAnd NameLoad Call AttributeLoad NameLoad attr Str body Assign NameStore IfExp NameLoad Str Str Assign NameStore Call NameLoad NameLoad NameLoad keyword AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad keyword NameLoad NameLoad With Call AttributeLoad NameLoad attr NameStore body Expr Call AttributeLoad NameLoad attr NameLoad Return Call AttributeLoad NameLoad attr NameLoad NameLoad keyword NameLoad keyword NameLoad orelse Return Call NameLoad NameLoad NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Str keyword Num Assign NameStore Dict AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Str Str Str Str Str Str Str Str Str FunctionDef arguments args NameParam defaults body Expr Str If Call NameLoad NameLoad TupleLoad NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad If Call NameLoad NameLoad AttributeLoad NameLoad attr body Assign NameStore AttributeLoad NameLoad attr If Call NameLoad NameLoad AttributeLoad NameLoad attr body Assign NameStore ListComp TupleLoad NameLoad Call NameLoad NameLoad comprehension TupleStore NameStore NameStore Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr Return ListComp BinOpMod Str NameLoad comprehension NameStore NameLoad If Call NameLoad NameLoad AttributeLoad NameLoad attr body Assign NameStore AttributeLoad NameLoad attr If Call NameLoad NameLoad AttributeLoad NameLoad attr body If AttributeLoad NameLoad attr body Return BinOpMod Str AttributeLoad NameLoad attr orelse Return Str If CompareIn NameLoad NameLoad body Return SubscriptLoad NameLoad Index NameLoad Raise Call NameLoad BinOpMod Str NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad Str Str Str kwarg body Expr Str If NameLoad body Assign NameStore BinOpAdd NameLoad Str If UnaryOpNot NameLoad body Assign NameStore Str Assert Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad Call AttributeLoad Str attr GeneratorExp BinOpMod Str Call NameLoad Call AttributeLoad NameLoad attr comprehension NameStore NameLoad attr Assign NameStore Str If CompareIn Str NameLoad body AugAssignAdd NameStore Str AugAssignAdd NameStore Str If NameLoad body Assign NameStore BinOpAdd NameLoad Str If Call AttributeLoad NameLoad attr Str body Assign NameStore BinOpAdd NameLoad Str Return Call AttributeLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad attr Str decorator_list Expr Str FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad kwarg body Expr Str Assign NameStore Str If UnaryOpNot NameLoad body Assign NameStore Call NameLoad NameLoad If CompareEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str body Assign NameStore Call NameLoad AttributeLoad NameLoad attr NameLoad keyword AttributeLoad NameLoad attr keyword SubscriptLoad Call AttributeLoad Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str Index Num keyword NameLoad Call NameLoad NameLoad orelse Raise Call NameLoad BinOpMod Str AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr With Call AttributeLoad AttributeLoad NameLoad attr attr NameStore body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr Return NameLoad decorator_list Call AttributeLoad NameLoad attr NameLoad Call NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad kwarg body Raise Call NameLoad Str decorator_list Call AttributeLoad NameLoad attr NameLoad TupleLoad Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad kwarg body If UnaryOpNot NameLoad body Assign NameStore Call NameLoad NameLoad If CompareEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str body Assign NameStore Call NameLoad AttributeLoad NameLoad attr NameLoad keyword SubscriptLoad Call AttributeLoad Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str Index Num Call NameLoad NameLoad orelse Raise Call NameLoad BinOpMod Str AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr With Call AttributeLoad AttributeLoad NameLoad attr attr NameStore body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr Return Call NameLoad NameLoad NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr NameLoad TupleLoad Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam NameParam defaults kwarg body Return Call NameLoad NameLoad Call NameLoad Call NameLoad Call NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr NameLoad NameLoad FunctionDef arguments args NameParam NameParam defaults kwarg body Expr Str Assign NameStore AttributeLoad NameLoad attr If CompareNotEq SubscriptLoad NameLoad Index Num Str body Assign NameStore BinOpMod Str TupleLoad SubscriptLoad AttributeLoad NameLoad attr Index Str AttributeLoad NameLoad attr If CompareEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str body Assign NameStore BinOpMod Str Dict Str Str NameLoad AttributeLoad NameLoad attr orelse Raise Call NameLoad BinOpMod Str AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr With Call AttributeLoad AttributeLoad NameLoad attr attr NameStore body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr TupleLoad Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam NameParam defaults kwarg body Expr Str If CompareNotEq AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Str body Raise Call NameLoad BinOpMod Str AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr Assign NameStore BinOpMod Str Dict Str Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr With Call AttributeLoad AttributeLoad NameLoad attr attr NameStore body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr TupleLoad Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam NameParam defaults kwarg body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad With Call NameLoad AttributeLoad NameLoad attr Str NameStore body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list Call AttributeLoad NameLoad attr NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam defaults kwarg body Assign NameStore Call AttributeLoad SubscriptLoad Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr Index Num attr Str Assign NameStore BinOpMod Str TupleLoad Call NameLoad Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call Call NameLoad AttributeLoad NameLoad attr NameLoad Return Call NameLoad NameLoad NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults Num kwarg body If Call NameLoad NameLoad Lambda arguments args defaults Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Str body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Str Str With Call NameLoad AttributeLoad NameLoad attr Str NameStore body Assign NameStore Call NameLoad NameLoad NameLoad For NameStore NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Str Call AttributeLoad Str attr NameLoad Return NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad TupleLoad NameLoad Call NameLoad NameLoad Import alias Assign NameStore Call AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults kwarg body Expr Str Assign NameStore Call NameLoad ListLoad Str Str Str Str Str Str Str Str Str If Call NameLoad NameLoad TupleLoad Call NameLoad NameLoad Call NameLoad NameLoad body With Call NameLoad NameLoad NameStore body Assign NameStore Call NameLoad Call NameLoad NameLoad AttributeLoad NameLoad attr orelse If Call NameLoad NameLoad TupleLoad Call NameLoad Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad body With Call NameLoad NameLoad NameStore body Assign NameStore Call NameLoad Call NameLoad NameLoad AttributeLoad NameLoad attr orelse If Call NameLoad NameLoad Call NameLoad NameLoad body Assign NameStore Call NameLoad Call NameLoad NameLoad orelse Assert Call NameLoad NameLoad NameLoad With Call NameLoad AttributeLoad NameLoad attr Str NameStore body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad GeneratorExp TupleLoad NameLoad Call NameLoad NameLoad NameLoad comprehension NameStore NameLoad Call NameLoad NameLoad NameLoad If CompareIs AttributeLoad NameLoad attr NameLoad body Assign SubscriptStore NameLoad Index Str Call AttributeLoad NameLoad attr NameLoad orelse Assign SubscriptStore NameLoad Index Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad GeneratorExp TupleLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr CompareIn NameLoad NameLoad Return NameLoad decorator_list Assign NameStore Dict Str Str Str Str NameLoad NameLoad NameLoad NameLoad Assign NameStore Str FunctionDef arguments args NameParam defaults kwarg body If CompareIn Str NameLoad body Assign NameStore SubscriptLoad NameLoad Slice Call NameLoad Str Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad NameLoad attr Assign NameStore Call NameLoad GeneratorExp TupleLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr CompareIsNot NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad TryExcept body Assign NameStore SubscriptLoad NameLoad Index SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num If CompareIn Str NameLoad body Assign NameStore Call NameLoad NameLoad Assign NameStore BinOpAdd SubscriptLoad Call AttributeLoad NameLoad attr Str Num Index Num Str handlers ExceptHandler type NameLoad body Assign NameStore Call NameLoad Call NameLoad NameLoad Return Call Call NameLoad NameLoad NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Str keyword Num FunctionDef arguments args NameParam NameParam defaults kwarg body Assign NameStore Call NameLoad Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad Return Call NameLoad NameLoad NameLoad NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad NameLoad TupleLoad NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad TupleLoad NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad TupleLoad Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad FunctionDef arguments args NameParam NameParam defaults kwarg body Raise Call NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam NameParam defaults kwarg body Raise Call NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad FunctionDef arguments args NameParam NameParam defaults kwarg body Raise Call NameLoad decorator_list Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad
N/A __future__ absolute_import division print_function toolz memoize merge partition_all multipledispatch MDNotImplementedError N/A re N/A os csv CSV json JSON JSONLines text TextFile N/A pandas pd N/A uuid N/A datashape N/A sqlalchemy sa datashape discover datashape coretypes ct collections namedtuple Iterator contextlib contextmanager ssh SSH sql metadata_of_engine utils tmpfile sample ignoring raises temp Temp append append convert convert chunks chunks resource resource directory Directory compatibility unicode N/A N/A ignoring ImportError N/A pywebhdfs.webhdfs PyWebHdfsClient pywebhdfs.errors FileNotFound _HDFS N/A object N/A N/A  Parent class for data on Hadoop File System

    Examples
    --------

    >>> HDFS(CSV)('/path/to/file.csv', host='54.91.255.255',
    ...           port=14000, user='hdfs')  # doctest: +SKIP

    Alternatively use resource strings

    >>> resource('hdfs://hdfs@54.91.255.255:/path/to/file.csv')  # doctest: +SKIP
     __init__ N/A N/A self N/A args kwargs N/A N/A hdfs N/A N/A kwargs get hdfs None N/A host N/A N/A kwargs get host None N/A user N/A N/A N/A kwargs get user_name N/A N/A kwargs get user N/A N/A kwargs get username None N/A port N/A str N/A N/A kwargs get port 14000 N/A N/A N/A hdfs N/A host port user N/A N/A hdfs N/A PyWebHdfsClient host host port N/A str port user_name user N/A N/A hdfs None N/A N/A N/A ValueError No HDFS credentials found.
Either supply a PyWebHdfsClient instance or keywords
   host=, port=, user= N/A N/A self hdfs hdfs N/A N/A N/A N/A self subtype __init__ self args kwargs N/A N/A HDFS N/A N/A cls N/A N/A N/A N/A type N/A HDFS(%s) N/A cls __name__ N/A _HDFS cls N/A subtype cls N/A memoize sample_hdfs_csv N/A N/A data length N/A 10000 N/A N/A sample N/A N/A N/A data hdfs read_file N/A N/A N/A data path lstrip / length length N/A N/A tmpfile N/A data canonical_extension fn N/A N/A N/A open fn w f N/A N/A N/A N/A f write sample N/A N/A fn N/A N/A N/A sample register _HDFS contextmanager discover_hdfs_file N/A N/A data N/A kwargs N/A N/A N/A sample data fn N/A N/A result N/A discover N/A N/A data subtype fn kwargs N/A result N/A N/A N/A discover register N/A HDFS JSON N/A N/A discover register N/A HDFS JSONLines N/A N/A discover register N/A HDFS TextFile discover_hdfs_csv N/A N/A data length N/A 10000 kwargs N/A N/A N/A sample data length length fn N/A N/A result N/A discover N/A CSV fn encoding N/A data encoding dialect N/A data dialect has_header N/A data has_header N/A result N/A N/A N/A discover register N/A HDFS CSV sample_hdfs_directory_csv N/A N/A data N/A kwargs N/A N/A files N/A N/A N/A data hdfs list_dir N/A N/A N/A data path lstrip / N/A one_file N/A N/A N/A data path / N/A N/A N/A N/A files N/A FileStatuses N/A FileStatus N/A 0 N/A pathSuffix N/A csv N/A N/A HDFS CSV one_file hdfs N/A data hdfs N/A N/A sample csv kwargs fn N/A N/A N/A fn N/A N/A N/A sample register N/A HDFS N/A Directory CSV contextmanager discover_hdfs_directory N/A N/A data length N/A 10000 kwargs N/A N/A N/A sample data length length fn N/A N/A o N/A N/A data container fn N/A data kwargs N/A result N/A discover o N/A result N/A N/A N/A discover register N/A HDFS N/A Directory CSV N/A 
Hive Tables
===========

Hive acts a bit differently from other databases that we interact with through
SQL.  As a result we need to special case a lot of code.

Foremost, a database is just a directory on HDFS holding something like a CSV
file (or Avro, Parquet, etc..)  As a result when we construct a Table we
actually have to know a lot about our CSV files (e.g. delimiter.)

This breaks the current odo model a bit because we usually create things with
`resource` and then `append` on to them.  Here we need to know both at the
same time.  Enter a convenient hack, a token for a proxy table
 N/A TableProxy N/A namedtuple TableProxy engine,name,stored_as N/A 
resource('hive://...::tablename') now gives us one of these.  The
subsequent call to `append` does the actual creation.

We're looking for better solutions.  For the moment, this works.
 resource_hive_table N/A N/A uri stored_as external dshape N/A TEXTFILE True None kwargs N/A N/A dshape N/A N/A dshape N/A N/A datashape dshape dshape N/A N/A uri table N/A N/A uri split :: N/A engine N/A resource uri N/A metadata N/A metadata_of_engine engine N/A N/A ignoring N/A N/A sa exc NoSuchTableError N/A N/A N/A N/A sa Table table metadata autoload True autoload_with engine N/A N/A dshape N/A N/A external N/A external N/A N/A kwargs get path N/A N/A table_type N/A external EXTERNAL N/A statement N/A create_hive_statement table dshape db_name N/A N/A engine url database stored_as stored_as table_type table_type kwargs N/A N/A N/A engine connect conn N/A N/A N/A N/A conn execute statement N/A N/A N/A sa Table table metadata autoload True autoload_with engine N/A N/A N/A TableProxy engine table stored_as N/A N/A N/A resource register hive://.+::.+ priority 16 N/A hive_types N/A N/A ct int8 N/A ct int16 N/A ct int32 N/A ct int64 N/A ct float32 N/A ct float64 N/A ct date_ N/A ct datetime_ N/A ct string N/A ct bool_ TINYINT SMALLINT INT BIGINT FLOAT DOUBLE DATE TIMESTAMP STRING BOOLEAN dshape_to_hive N/A N/A ds N/A N/A N/A  Convert datashape measure to Hive dtype string

    >>> dshape_to_hive('var * {name: string, balance: int32}')
    ['name  STRING', 'balance  INT']
    >>> dshape_to_hive('int16')
    'SMALLINT'
    >>> dshape_to_hive('?int32')  # Ignore option types
    'INT'
    >>> dshape_to_hive('string[256]')
    'VARCHAR(256)'
     N/A N/A isinstance ds N/A str unicode N/A N/A ds N/A N/A datashape dshape ds N/A N/A isinstance ds N/A ct DataShape N/A N/A ds N/A ds measure N/A N/A isinstance ds N/A ct Record N/A N/A columns N/A N/A name N/A dshape_to_hive typ N/A N/A name typ N/A zip N/A N/A ds measure names N/A N/A ds measure types N/A N/A N/A %s  %s col N/A col columns N/A N/A isinstance ds N/A ct Option N/A N/A ds N/A ds ty N/A N/A isinstance ds N/A ct String N/A N/A N/A ds fixlen N/A N/A N/A VARCHAR(%d) N/A ds fixlen N/A N/A STRING N/A N/A ds hive_types N/A N/A N/A hive_types N/A ds N/A N/A NotImplementedError N/A No Hive dtype known for %s ds N/A create_hive_statement N/A N/A tbl_name dshape path table_type db_name stored_as N/A None default TEXTFILE kwargs N/A N/A  Generic CREATE TABLE statement for hive

    Parameters
    ----------

    tbl_name : string
        Specifies table name "mytable"
    dshape : DataShape
        Datashape of the desired table
    path : string (optional)
        Location of data
    table_type : string (optional)
        Table Modifier like EXTERNAL or LOCAL
    db_name : string
        Specifies database name.  Defaults to "default"
    stored_as: string
        Target storage format like TEXTFILE or PARQUET

    **kwargs: keyword arguments dict
        CSV dialect with keys delimiter, has_header, etc.

    Example
    -------

    >>> from datashape import dshape
    >>> ds = dshape('var * {name: string, balance: int64, when: datetime}')
    >>> print(create_hive_statement('accounts', ds, delimiter=','))  # doctest: +NORMALIZE_WHITESPACE
    CREATE  TABLE default.accounts (
            name  STRING,
         balance  BIGINT,
            when  TIMESTAMP
    )
    ROW FORMAT DELIMITED
        FIELDS TERMINATED BY ','
    STORED AS TEXTFILE

    >>> print(create_hive_statement('accounts', ds, delimiter=',',
    ...         has_header=True, path='/data/accounts/', table_type='EXTERNAL'))  # doctest: +NORMALIZE_WHITESPACE
    CREATE EXTERNAL TABLE default.accounts (
            name  STRING,
         balance  BIGINT,
            when  TIMESTAMP
    )
    ROW FORMAT DELIMITED
        FIELDS TERMINATED BY ','
    STORED AS TEXTFILE
    LOCATION '/data/accounts/'
    TBLPROPERTIES ("skip.header.line.count"="1")

    >>> print(create_hive_statement('accounts', ds, stored_as='PARQUET'))  # doctest: +NORMALIZE_WHITESPACE
    CREATE  TABLE default.accounts (
            name  STRING,
         balance  BIGINT,
            when  TIMESTAMP
    )
    STORED AS PARQUET
     N/A db_name N/A N/A db_name N/A db_name . N/A N/A table_type N/A N/A table_type N/A N/A isinstance N/A dshape measure N/A ct Record N/A columns N/A dshape_to_hive dshape N/A column_text N/A N/A N/A N/A ,
     join N/A N/A %20s  %s N/A tuple N/A N/A col split N/A col columns lstrip N/A statement 
        CREATE {table_type} TABLE {db_name}{tbl_name} (
            {column_text}
        )
         N/A N/A delimiter kwargs N/A N/A statement 
        ROW FORMAT DELIMITED
            FIELDS TERMINATED BY '{delimiter}'
         N/A statement 
        STORED AS {stored_as}
         N/A path N/A N/A statement N/A statement 
        LOCATION '{path}'
         N/A N/A N/A kwargs get has_header N/A N/A statement N/A statement 
        TBLPROPERTIES ("skip.header.line.count"="1") N/A N/A N/A N/A N/A statement format N/A merge kwargs N/A locals strip N/A N/A 
Load Data from HDFS or SSH into Hive
====================================

We push types like HDFS(CSV) and HDFS(Directory(CSV)) into Hive tables.  This
requires that we bring a bit of the CSV file locally, inspect it (sniff for csv
dialect), generate the appropriate CREATE EXTERNAL TABLE command, and then
execute.
 create_new_hive_table_from_csv N/A N/A tbl data dshape path N/A None None kwargs N/A N/A 
    Create new Hive table from directory of CSV files on HDFS

    Actually executes command.
     N/A table_type EXTERNAL N/A N/A dshape N/A N/A dshape N/A discover data N/A N/A N/A N/A N/A tbl engine dialect name hive N/A N/A statement N/A create_hive_statement N/A tbl name dshape path N/A data path db_name N/A N/A N/A N/A str N/A N/A tbl engine url split / N/A -1 table_type table_type N/A dialect_of data N/A N/A N/A NotImplementedError N/A Don't know how to migrate directory of csvs on HDFS to database of dialect %s N/A N/A N/A tbl engine dialect name N/A N/A N/A N/A tbl engine connect conn N/A N/A N/A N/A conn execute statement N/A metadata N/A metadata_of_engine N/A tbl engine N/A tbl2 N/A N/A sa Table N/A tbl name metadata autoload True autoload_with N/A tbl engine N/A tbl2 N/A N/A N/A append register TableProxy N/A HDFS N/A Directory CSV create_new_hive_table_from_csv_file N/A N/A tbl data dshape path N/A None None kwargs N/A N/A N/A ValueError Can not create a new Hive table from a single CSV file on HDFS.
Instead try loading a complete directory or base your data outside of HDFS N/A N/A N/A append register TableProxy N/A N/A HDFS CSV N/A Temp N/A HDFS CSV append_remote_csv_to_new_table N/A N/A tbl data dshape N/A None kwargs N/A N/A N/A dshape N/A N/A dshape N/A discover data N/A N/A N/A N/A N/A tbl engine dialect name hive N/A N/A statement N/A create_hive_statement N/A tbl name dshape db_name N/A N/A N/A N/A str N/A N/A tbl engine url split / N/A -1 N/A dialect_of data N/A N/A N/A NotImplementedError N/A Don't know how to migrate directory of csvs on Local disk to database of dialect %s N/A N/A N/A tbl engine dialect name N/A N/A N/A N/A tbl engine connect conn N/A N/A N/A N/A conn execute statement N/A metadata N/A metadata_of_engine N/A tbl engine N/A tbl2 N/A N/A sa Table N/A tbl name metadata autoload True autoload_with N/A tbl engine N/A N/A append tbl2 data kwargs N/A N/A N/A append register TableProxy N/A N/A SSH CSV N/A SSH N/A Directory CSV append_anything_to_tableproxy N/A N/A tbl data N/A kwargs N/A N/A N/A append tbl N/A convert N/A Temp N/A SSH CSV data kwargs kwargs N/A N/A N/A append register TableProxy object append_remote_csv_to_table N/A N/A tbl csv N/A kwargs N/A N/A 
    Load Remote data into existing Hive table
     N/A path N/A csv path N/A N/A N/A path N/A 0 / N/A N/A path N/A /home/%s/%s N/A N/A N/A csv auth N/A username N/A csv path N/A N/A N/A N/A N/A tbl bind dialect name hive N/A N/A statement N/A LOAD DATA LOCAL INPATH "%(path)s" INTO TABLE %(tablename)s N/A path tablename path N/A tbl name N/A N/A N/A NotImplementedError N/A Don't know how to migrate csvs on remote disk to database of dialect %s N/A N/A N/A tbl engine dialect name N/A N/A N/A N/A tbl bind connect conn N/A N/A N/A N/A conn execute statement N/A tbl N/A N/A N/A append register N/A sa Table N/A N/A SSH CSV N/A SSH N/A Directory CSV append_hdfs_csv_to_table N/A N/A tbl csv N/A kwargs N/A N/A 
    Load Remote data into existing Hive table
     N/A N/A N/A N/A N/A tbl bind dialect name hive N/A N/A N/A NotImplementedError N/A Don't know how to migrate csvs on remote disk to database of dialect %s N/A N/A N/A tbl engine dialect name N/A statement N/A LOAD DATA INPATH "%(path)s" INTO TABLE %(tablename)s N/A path tablename N/A csv path N/A tbl name N/A N/A N/A N/A tbl bind connect conn N/A N/A N/A N/A conn execute statement N/A tbl N/A N/A N/A append register N/A sa Table N/A N/A HDFS CSV N/A HDFS N/A Directory CSV N/A Temp N/A HDFS CSV append_hdfs_file_to_local N/A N/A target source N/A kwargs N/A N/A text N/A N/A N/A source hdfs read_file N/A N/A N/A source path lstrip / kwargs N/A N/A open N/A target path w f N/A N/A N/A N/A f write text N/A target N/A N/A N/A append register TextFile N/A HDFS TextFile N/A N/A append register JSONLines N/A HDFS JSONLines N/A N/A append register JSON N/A HDFS JSON N/A N/A append register CSV N/A HDFS CSV convert_hdfs_file_to_temp_local N/A N/A source N/A kwargs N/A N/A ext N/A N/A N/A N/A N/A N/A os path splitext N/A source path N/A 1 strip . N/A fn N/A .%s.%s N/A N/A str N/A N/A uuid uuid1 ext N/A tmp N/A N/A Temp N/A source subtype fn N/A N/A append tmp source kwargs N/A N/A N/A convert register N/A Temp TextFile N/A HDFS TextFile N/A N/A convert register N/A Temp JSONLines N/A HDFS JSONLines N/A N/A convert register N/A Temp JSON N/A HDFS JSON N/A N/A convert register N/A Temp CSV N/A HDFS CSV append_local_file_to_hdfs N/A N/A target source blocksize N/A 100000 kwargs N/A N/A N/A raises FileNotFound N/A N/A N/A N/A N/A N/A N/A target hdfs list_dir N/A N/A N/A target path lstrip / N/A N/A N/A N/A N/A target hdfs create_file N/A N/A N/A target path lstrip / N/A N/A open N/A source path r f N/A N/A blocks N/A partition_all blocksize f N/A block blocks N/A N/A N/A N/A N/A target hdfs append_file N/A N/A N/A target path lstrip / N/A N/A join block N/A target N/A N/A N/A append register N/A HDFS TextFile TextFile N/A N/A append register N/A HDFS JSONLines JSONLines N/A N/A append register N/A HDFS JSON JSON N/A N/A append register N/A HDFS CSV N/A CSV N/A Temp CSV N/A csv N/A sniffer N/A N/A csv Sniffer dialect_of N/A N/A data N/A kwargs N/A N/A  CSV dialect of a CSV file stored in SSH, HDFS, or a Directory.  N/A keys N/A set N/A delimiter doublequote escapechar lineterminator quotechar quoting skipinitialspace strict has_header N/A N/A isinstance data N/A N/A HDFS CSV N/A SSH CSV N/A N/A N/A sample data fn N/A N/A d N/A dialect_of N/A CSV fn N/A data dialect N/A N/A N/A isinstance data N/A N/A HDFS N/A Directory CSV N/A SSH N/A Directory CSV N/A N/A N/A sample data fn N/A N/A d N/A dialect_of N/A CSV fn N/A data kwargs N/A N/A N/A isinstance data N/A Directory CSV N/A N/A d N/A dialect_of N/A next data N/A N/A N/A isinstance data CSV N/A N/A open N/A data path r f N/A N/A text N/A N/A f read N/A result N/A dict N/A d N/A N/A sniffer sniff text N/A d N/A dict N/A N/A k N/A getattr d k N/A k keys N/A hasattr d k N/A N/A N/A data has_header None N/A N/A N/A d N/A has_header N/A N/A sniffer has_header text N/A N/A N/A d N/A has_header N/A data has_header N/A N/A N/A d update N/A data dialect N/A N/A N/A d update kwargs N/A d N/A dict N/A N/A k v N/A N/A k v N/A N/A d items N/A k keys N/A d N/A N/A types_by_extension N/A csv json txt log CSV JSONLines TextFile TextFile N/A hdfs_pattern (((?P<user>[a-zA-Z]\w*)@)?(?P<host>[\w.-]*)?(:(?P<port>\d+))?:)?(?P<path>[/\w.*-]+) resource_hdfs N/A N/A uri N/A kwargs N/A N/A N/A hdfs:// uri N/A N/A uri N/A uri N/A N/A len hdfs:// N/A d N/A N/A N/A N/A re match hdfs_pattern uri groupdict N/A d N/A dict N/A N/A k v N/A N/A k v N/A N/A d items N/A v None N/A path N/A N/A d pop path N/A N/A N/A kwargs update d N/A N/A N/A subtype N/A types_by_extension N/A N/A N/A N/A path split . N/A -1 N/A N/A * path N/A N/A subtype N/A Directory subtype N/A path N/A N/A N/A N/A path rsplit / 1 N/A 0 / N/A N/A N/A KeyError N/A N/A subtype N/A type N/A resource path N/A N/A N/A HDFS subtype path kwargs N/A N/A N/A resource register hdfs://.* priority 16 append_object_to_hdfs N/A N/A target source N/A kwargs N/A N/A tmp N/A convert N/A Temp N/A target subtype source kwargs N/A N/A append target tmp kwargs N/A N/A N/A append register N/A HDFS TextFile N/A Iterator object N/A N/A append register N/A HDFS JSON N/A list object N/A N/A append register N/A HDFS CSV N/A N/A chunks N/A pd DataFrame N/A pd DataFrame object append_remote_file_to_hdfs N/A N/A target source N/A kwargs N/A N/A N/A MDNotImplementedError N/A N/A N/A append register N/A HDFS TextFile N/A SSH TextFile N/A N/A append register N/A HDFS JSONLines N/A SSH JSONLines N/A N/A append register N/A HDFS JSON N/A SSH JSON N/A N/A append register N/A HDFS CSV N/A SSH CSV append_hdfs_file_to_hdfs_file N/A N/A target source N/A kwargs N/A N/A N/A MDNotImplementedError N/A N/A N/A append register N/A HDFS TextFile N/A HDFS TextFile N/A N/A append register N/A HDFS JSONLines N/A HDFS JSONLines N/A N/A append register N/A HDFS JSON N/A HDFS JSON N/A N/A append register N/A HDFS CSV N/A HDFS CSV append_hdfs_file_to_remote N/A N/A target source N/A kwargs N/A N/A N/A MDNotImplementedError N/A N/A N/A append register N/A SSH TextFile N/A HDFS TextFile N/A N/A append register N/A SSH JSONLines N/A HDFS JSONLines N/A N/A append register N/A SSH JSON N/A HDFS JSON N/A N/A append register N/A SSH CSV N/A HDFS CSV