Module ImportFrom alias ImportFrom alias ImportFrom alias Import alias Import alias ImportFrom alias ImportFrom alias alias alias ImportFrom alias ImportFrom alias ClassDef bases NameLoad body Expr Str Assign NameStore TupleLoad Str Str Str Str Assign NameStore Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad Str Str Str vararg kwarg body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr BoolOpOr NameLoad Dict Assign AttributeStore NameLoad attr BoolOpOr NameLoad Dict Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Str Assign AttributeStore NameLoad attr Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults body If CompareIn NameLoad AttributeLoad NameLoad attr body Return Dict Assign NameStore Dict Assign SubscriptStore NameLoad Index TupleLoad NameLoad Str Str If CompareIn NameLoad ListLoad Str Str Str Str Str body Assign SubscriptStore NameLoad Index TupleLoad NameLoad Str Str Assign SubscriptStore NameLoad Index TupleLoad NameLoad Str Str Assign SubscriptStore NameLoad Index TupleLoad NameLoad Str Str Assign SubscriptStore NameLoad Index TupleLoad NameLoad Str Str orelse If CompareEq NameLoad Str body Assign SubscriptStore NameLoad Index TupleLoad NameLoad Str Str Assign SubscriptStore NameLoad Index TupleLoad NameLoad Str Str orelse If CompareIn NameLoad ListLoad Str body Assign SubscriptStore NameLoad Index TupleLoad NameLoad Str Str Assign SubscriptStore NameLoad Index TupleLoad NameLoad Str Str Return DictComp NameLoad Call AttributeLoad NameLoad attr keyword NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Assign NameStore Call NameLoad keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Assign NameStore DictComp AttributeLoad NameLoad attr AttributeLoad NameLoad attr comprehension NameStore AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Dict TupleLoad Str Str Str For TupleStore NameStore NameStore Call NameLoad Call AttributeLoad NameLoad attr body Assign NameStore Dict If AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If CompareIs NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad Str attr ListComp BinOpAdd BinOpAdd BinOpAdd BinOpAdd NameLoad Str SubscriptLoad NameLoad Index Num Str SubscriptLoad NameLoad Index Num comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore ListComp Call AttributeLoad Str attr NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad Str attr NameLoad Assign NameStore Call AttributeLoad Str attr Call NameLoad Assign NameStore Call NameLoad keyword AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpAdd Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad NameLoad attr BinOpAdd Str Call NameLoad NameLoad If UnaryOpNot NameLoad body Raise Call NameLoad Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad Str attr Call NameLoad If Call AttributeLoad NameLoad attr NameLoad body Assign NameStore Call AttributeLoad Str attr Call NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Str Assign NameStore ListComp BinOpAdd TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad TupleLoad SubscriptLoad SubscriptLoad NameLoad Index Num Index Num SubscriptLoad SubscriptLoad NameLoad Index Num Index Num SubscriptLoad NameLoad Index Num comprehension NameStore Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr keyword Str keyword NameLoad keyword ListLoad Str Str Str Str Str Str Str decorator_list decorator_list
N/A builtins str builtins zip collections OrderedDict N/A json N/A logging airflow.exceptions AirflowException airflow.hooks PrestoHook HiveMetastoreHook MySqlHook airflow.models BaseOperator airflow.utils.decorators apply_defaults HiveStatsCollectionOperator N/A BaseOperator N/A N/A 
    Gathers partition statistics using a dynamically generated Presto
    query, inserts the stats into a MySql table with this format. Stats
    overwrite themselves if you rerun the same date/partition.

    ``
    CREATE TABLE hive_stats (
        ds VARCHAR(16),
        table_name VARCHAR(500),
        metric VARCHAR(200),
        value BIGINT
    );
    ``

    :param table: the source table, in the format ``database.table_name``
    :type table: str
    :param partition: the source partition
    :type partition: dict of {col:value}
    :param extra_exprs: dict of expression to run against the table where
        keys are metric names and values are Presto compatible expressions
    :type extra_exprs: dict
    :param col_blacklist: list of columns to blacklist, consider
        blacklisting blobs, large json columns, ...
    :type col_blacklist: list
    :param assignment_func: a function that receives a column name and
        a type, and returns a dict of metric names and an Presto expressions.
        If None is returned, the global defaults are applied. If an
        empty dictionary is returned, no stats are computed for that
        column.
    :type assignment_func: function
     N/A template_fields N/A table partition ds dttm N/A ui_color #aff7a6 __init__ N/A N/A self table partition extra_exprs col_blacklist assignment_func metastore_conn_id presto_conn_id mysql_conn_id N/A None None None metastore_default presto_default airflow_db args kwargs N/A N/A N/A N/A N/A super HiveStatsCollectionOperator self __init__ args kwargs N/A N/A self table table N/A N/A self partition partition N/A N/A self extra_exprs N/A extra_exprs N/A N/A N/A self col_blacklist N/A col_blacklist N/A N/A N/A self metastore_conn_id metastore_conn_id N/A N/A self presto_conn_id presto_conn_id N/A N/A self mysql_conn_id mysql_conn_id N/A N/A self assignment_func assignment_func N/A N/A self ds {{ ds }} N/A N/A self dttm {{ execution_date.isoformat() }} N/A apply_defaults get_default_exprs N/A N/A self col col_type N/A N/A N/A N/A col N/A self col_blacklist N/A N/A N/A N/A d N/A N/A N/A d N/A N/A col non_null COUNT({col}) N/A N/A col_type N/A double int bigint float double N/A N/A N/A d N/A N/A col sum SUM({col}) N/A N/A d N/A N/A col min MIN({col}) N/A N/A d N/A N/A col max MAX({col}) N/A N/A d N/A N/A col avg AVG({col}) N/A N/A N/A col_type boolean N/A N/A N/A d N/A N/A col true SUM(CASE WHEN {col} THEN 1 ELSE 0 END) N/A N/A d N/A N/A col false SUM(CASE WHEN NOT {col} THEN 1 ELSE 0 END) N/A N/A N/A col_type N/A string N/A N/A N/A d N/A N/A col len SUM(CAST(LENGTH({col}) AS BIGINT)) N/A N/A d N/A N/A col approx_distinct APPROX_DISTINCT({col}) N/A N/A k N/A N/A v format col col N/A N/A k v N/A N/A d items N/A execute N/A N/A self context N/A None N/A N/A metastore N/A HiveMetastoreHook metastore_conn_id N/A self metastore_conn_id N/A table N/A N/A metastore get_table table_name N/A self table N/A field_types N/A N/A col name N/A col type N/A col N/A N/A table sd cols N/A exprs N/A N/A count COUNT(*) N/A N/A col col_type N/A list N/A N/A field_types items N/A N/A d N/A N/A N/A self assignment_func N/A N/A d N/A N/A self assignment_func col col_type N/A N/A d None N/A N/A d N/A N/A self get_default_exprs col col_type N/A N/A d N/A N/A self get_default_exprs col col_type N/A N/A N/A exprs update d N/A N/A N/A exprs update N/A self extra_exprs N/A exprs N/A OrderedDict exprs N/A exprs_str N/A N/A ,
         join N/A N/A N/A N/A N/A v  AS  N/A k N/A 0 __ N/A k N/A 1 N/A N/A k v N/A N/A exprs items N/A where_clause N/A N/A N/A {0} = '{1}' format k v N/A N/A k v N/A N/A N/A self partition items N/A where_clause N/A N/A  AND
         join where_clause N/A sql N/A N/A 
        SELECT
            {exprs_str}
        FROM {self.table}
        WHERE
            {where_clause};
         format N/A locals N/A hook N/A PrestoHook presto_conn_id N/A self presto_conn_id N/A N/A N/A logging info N/A Executing SQL check:  sql N/A row N/A N/A hook get_first hql sql N/A N/A N/A logging info N/A Record:  N/A str row N/A N/A row N/A N/A N/A AirflowException The query returned None N/A part_json N/A N/A json dumps N/A self partition sort_keys True N/A N/A N/A logging info Deleting rows from previous runs if they exist N/A mysql N/A MySqlHook N/A self mysql_conn_id N/A sql N/A N/A 
        SELECT 1 FROM hive_stats
        WHERE
            table_name='{self.table}' AND
            partition_repr='{part_json}' AND
            dttm='{self.dttm}'
        LIMIT 1;
         format N/A locals N/A N/A N/A mysql get_records sql N/A N/A sql N/A N/A 
            DELETE FROM hive_stats
            WHERE
                table_name='{self.table}' AND
                partition_repr='{part_json}' AND
                dttm='{self.dttm}';
             format N/A locals N/A N/A N/A mysql run sql N/A N/A N/A logging info Pivoting and loading cells into the Airflow db N/A rows N/A N/A N/A N/A self ds N/A self dttm N/A self table part_json N/A N/A N/A r N/A 0 N/A 0 N/A N/A r N/A 0 N/A 1 N/A r N/A 1 N/A r N/A zip exprs row N/A N/A N/A mysql insert_rows table hive_stats rows rows target_fields N/A ds dttm table_name partition_repr col metric value N/A N/A