Module Import alias Import alias Import alias ImportFrom alias alias ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias alias ImportFrom alias alias Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr Assign NameStore ListLoad Str Str Str Str Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore Call AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr ClassDef bases NameLoad body Expr Str Assign NameStore NameLoad Assign NameStore BinOpAdd AttributeLoad NameLoad attr TupleLoad Call NameLoad Str keyword Str keyword Str keyword Str keyword Str Call NameLoad Str keyword Str keyword Str keyword Str keyword Str keyword Str FunctionDef arguments args NameParam defaults vararg kwarg body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Index Str If UnaryOpNot NameLoad body Assign NameStore Call AttributeLoad BinOpSub Call AttributeLoad NameLoad attr Call NameLoad keyword Num attr Str Assign NameStore IfExp NameLoad SubscriptLoad NameLoad Index Num Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore TupleLoad Str Str Str Str Str Assign NameStore ListLoad For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad BinOpMod Str NameLoad If CompareNotEq Call NameLoad NameLoad NameLoad NameLoad body Raise Call NameLoad BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad Expr Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad attr Assign NameStore Dict Assign NameStore Call NameLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad attr keyword AttributeLoad NameLoad attr attr Str Str Assign NameStore Num For TupleStore NameStore NameStore NameLoad body With Call AttributeLoad NameLoad attr NameLoad keyword Str NameStore body For NameStore NameLoad body AugAssignAdd NameStore Num If BoolOpAnd NameLoad CompareEq BinOpMod NameLoad Num Num body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Assign NameStore Call AttributeLoad SubscriptLoad NameLoad Slice Num attr NameLoad If BoolOpOr BoolOpAnd CompareEq NameLoad Str CompareNotEq Call NameLoad NameLoad Num BoolOpAnd CompareNotEq NameLoad Str CompareNotEq Call NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Continue If CompareEq NameLoad Str body Assign TupleStore NameStore NameStore NameStore NameStore NameStore NameStore NameLoad orelse Assign TupleStore NameStore NameStore NameStore NameStore NameStore NameLoad Assign NameStore Call AttributeLoad NameLoad attr If NameLoad body Expr Call AttributeLoad NameLoad attr If CompareIn NameLoad ListLoad Str Str Str Str Str Str body Assign NameStore NameLoad TryExcept body Assign NameStore Call NameLoad NameLoad If NameLoad body Assign NameStore Call NameLoad NameLoad handlers ExceptHandler type NameLoad body Continue If BoolOpAnd NameLoad CompareNotEq BinOpBitAnd NameLoad Num Num body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Continue If BoolOpAnd NameLoad CompareIn NameLoad NameLoad body Assign NameStore SubscriptLoad NameLoad Index NameLoad orelse Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr keyword Call AttributeLoad NameLoad attr Continue If CompareIn NameLoad NameLoad body Assign NameStore SubscriptLoad NameLoad Index NameLoad orelse Assign NameStore Call NameLoad keyword NameLoad keyword NameLoad keyword Num Assign SubscriptStore NameLoad Index NameLoad NameLoad If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad AugAssignAdd AttributeStore NameLoad attr NameLoad orelse If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad orelse If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad orelse If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad orelse If CompareEq NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr Num For NameStore Call AttributeLoad NameLoad attr body Expr Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str BinOpAdd NameLoad Num Expr Call AttributeLoad NameLoad attr BinOpMod Str BinOpSub Call AttributeLoad NameLoad attr NameLoad For TupleStore NameStore NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr keyword NameLoad Expr Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad NameLoad Slice Num Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str If CompareIn NameLoad NameLoad body Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Expr Str If BoolOpOr CompareNotIn NameLoad NameLoad UnaryOpNot Call AttributeLoad NameLoad attr NameLoad NameLoad body Return If CompareIs AttributeLoad NameLoad attr NameLoad body Assign AttributeStore NameLoad attr Dict Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Dict Expr Call AttributeLoad AttributeLoad NameLoad attr attr Dict NameLoad Call NameLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str If CompareIn Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Str Str If Call AttributeLoad NameLoad attr NameLoad NameLoad body Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str FunctionDef arguments args NameParam defaults body Expr Str Return CompareLt Call NameLoad Call AttributeLoad NameLoad attr NameLoad BinOpPow Num Num decorator_list If Call NameLoad NameLoad body Return Assign NameStore Call NameLoad Call NameLoad Call NameLoad Call AttributeLoad NameLoad attr keyword Lambda arguments args NameParam defaults SubscriptLoad NameLoad Index Num While UnaryOpNot Call NameLoad NameLoad body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Delete SubscriptDel NameLoad Index NameLoad decorator_list decorator_list
N/A N/A codecs N/A json N/A re datetime datetime timedelta optparse make_option os path unlink django.conf settings django.core.management.base BaseCommand CommandError N/A commonware.log olympia amo olympia.addons.models Addon olympia.stats.models update_inc UpdateCount N/A get_date_from_file save_stats_to_file N/A log N/A N/A N/A commonware log getLogger adi.updatecountsfromfile N/A LOCALE_REGEX N/A N/A re compile ^[a-z]{2,3}      # General: fr, en, dsb,...
                              (-[A-Z]{2,3})?   # Region: -US, -GB, ...
                              (-[a-z]{2,12})?$ # Locality: -valencia, -diiste
                           N/A re VERBOSE N/A VALID_STATUSES N/A userDisabled,incompatible userEnabled Unknown userDisabled userEnabled,incompatible N/A VALID_APP_GUIDS N/A N/A N/A amo APP_GUIDS keys N/A APPVERSION_REGEX N/A N/A re compile ^[0-9]{1,3}                # Major version: 2, 35
        \.[0-9]{1,3}([ab][0-9])?   # Minor version + alpha or beta: .0a1, .0b2
        (\.[0-9]{1,3})?$           # Patch version: .1, .23
     N/A re VERBOSE Command N/A BaseCommand N/A N/A Process hive results stored in different files and store them in the db.

    Usage:
    ./manage.py update_counts_from_file <folder> --date=YYYY-MM-DD

    If no date is specified, the default is the day before.
    If not folder is specified, the default is `hive_results/<YYYY-MM-DD>/`.
    This folder will be located in `<settings.NETAPP_STORAGE>/tmp`.

    Five files are processed:
    - update_counts_by_version.hive
    - update_counts_by_status.hive
    - update_counts_by_app.hive
    - update_counts_by_os.hive
    - update_counts_by_locale.hive

    Each file has the following cols:
    - date
    - addon guid
    - data: the data grouped on (eg version, status...).
    - count
    - update type

    For the "app" file, the "data" col is in fact two cols: the application
    guid and the application version.

     N/A help __doc__ N/A option_list N/A N/A BaseCommand option_list N/A N/A make_option --date action store type string dest date help Date in the YYYY-MM-DD format. N/A make_option --separator action store type string default dest separator help Field separator in file. handle N/A N/A self N/A args options N/A N/A start N/A N/A datetime now N/A day N/A options N/A date N/A N/A day N/A N/A day N/A N/A N/A N/A N/A datetime now N/A timedelta days 1 strftime %Y-%m-%d N/A folder N/A args N/A args N/A 0 hive_results N/A folder N/A N/A path join N/A settings TMP_PATH folder day N/A sep N/A options N/A separator N/A groups N/A version status app os locale N/A group_filepaths N/A N/A group groups N/A N/A filepath N/A N/A path join folder N/A update_counts_by_%s.hive group N/A N/A N/A get_date_from_file filepath sep day N/A N/A N/A CommandError N/A %s file contains data for another day filepath N/A N/A N/A group_filepaths append N/A group filepath N/A N/A N/A N/A N/A N/A UpdateCount objects filter date day delete N/A update_counts N/A N/A guids_to_addon N/A dict N/A N/A N/A N/A N/A N/A N/A Addon objects exclude guid__isnull True exclude type N/A amo ADDON_PERSONA values_list guid id N/A index -1 N/A N/A group filepath group_filepaths N/A N/A N/A N/A codecs open filepath encoding utf8 results_file N/A N/A line results_file N/A N/A index 1 N/A N/A index N/A N/A index 1000000 0 N/A N/A N/A N/A log info N/A Processed %s lines index N/A splitted N/A N/A N/A line N/A -1 split sep N/A N/A N/A N/A group app N/A N/A len splitted 6 N/A N/A group app N/A N/A len splitted 5 N/A N/A N/A N/A log debug N/A Badly formatted row: %s line N/A N/A N/A group app N/A N/A N/A day addon_guid app_id app_ver count update_type splitted N/A N/A N/A day addon_guid data count update_type splitted N/A addon_guid N/A N/A addon_guid strip N/A update_type N/A N/A N/A N/A update_type strip N/A N/A update_type N/A 0 NULL None \N %UPDATE_TYPE% N/A N/A update_type None N/A N/A N/A count N/A int count N/A update_type N/A N/A update_type N/A int update_type N/A N/A N/A ValueError N/A N/A N/A N/A update_type N/A N/A update_type 31 16 N/A N/A N/A N/A log debug N/A Update type doesn't add to 16: %s update_type N/A N/A N/A addon_guid N/A addon_guid guids_to_addon N/A N/A addon_id N/A guids_to_addon N/A addon_guid N/A N/A N/A N/A log debug N/A N/A Addon {guid} doesn't exist. format guid N/A N/A addon_guid strip N/A N/A N/A addon_guid update_counts N/A N/A uc N/A update_counts N/A addon_guid N/A N/A uc N/A UpdateCount date day addon_id addon_id count 0 N/A N/A update_counts N/A addon_guid uc N/A N/A group version N/A N/A N/A N/A self update_version uc data count N/A N/A uc count count N/A N/A N/A group status N/A N/A N/A N/A self update_status uc data count N/A N/A N/A group app N/A N/A N/A N/A self update_app uc app_id app_ver count N/A N/A N/A group os N/A N/A N/A N/A self update_os uc data count N/A N/A N/A group locale N/A N/A N/A N/A self update_locale uc data count N/A N/A addon_guid update_count N/A N/A update_counts iteritems N/A N/A N/A N/A self trim_field N/A update_count locales N/A N/A N/A self trim_field N/A update_count versions N/A N/A N/A N/A UpdateCount objects bulk_create N/A N/A update_counts values 100 N/A udate_count N/A N/A update_counts values N/A N/A N/A save_stats_to_file update_count N/A N/A N/A log info N/A Processed a total of %s lines N/A index 1 N/A N/A N/A log debug N/A Total processing time: %s N/A N/A N/A datetime now start N/A N/A _ filepath group_filepaths N/A N/A N/A N/A log debug N/A N/A Deleting {path} format path filepath N/A N/A unlink filepath N/A update_version N/A N/A self update_count version count N/A N/A N/A Update the versions on the update_count with the given version. N/A version N/A version N/A 32 N/A N/A update_count versions N/A update_inc N/A update_count versions version count N/A update_status N/A N/A self update_count status count N/A N/A N/A Update the statuses on the update_count with the given status. N/A N/A status VALID_STATUSES N/A N/A N/A update_count statuses N/A update_inc N/A update_count statuses status count N/A update_app N/A N/A self update_count app_id app_ver count N/A N/A N/A Update the applications on the update_count with the given data. N/A N/A N/A app_id VALID_APP_GUIDS N/A N/A N/A re match APPVERSION_REGEX app_ver N/A N/A N/A N/A N/A update_count applications None N/A N/A N/A update_count applications N/A N/A app N/A N/A N/A update_count applications get app_id N/A N/A N/A N/A N/A update_count applications update N/A app_id N/A update_inc app app_ver count N/A update_os N/A N/A self update_count os count N/A N/A N/A Update the OSes on the update_count with the given OS. N/A N/A N/A N/A os lower N/A amo PLATFORM_DICT N/A N/A N/A update_count oses N/A update_inc N/A update_count oses os count N/A update_locale N/A N/A self update_count locale count N/A N/A N/A Update the locales on the update_count with the given locale. N/A locale N/A N/A locale replace _ - N/A N/A N/A re match LOCALE_REGEX locale N/A N/A N/A update_count locales N/A update_inc N/A update_count locales locale count N/A trim_field N/A N/A self field N/A N/A N/A Trim (in-place) the dict provided, keeping the most used items.

        The "locales" and "versions" fields are dicts which have the locale
        or version as the key, and the count as the value.

         fits N/A N/A field N/A N/A N/A Does the json version of the field fits in the db TEXT field? N/A N/A N/A len N/A N/A json dumps field N/A 2 16 N/A N/A N/A fits field N/A N/A N/A values N/A list N/A reversed N/A sorted N/A N/A field items key N/A N/A N/A v N/A N/A v N/A 1 N/A N/A N/A fits field N/A N/A N/A key count N/A N/A values pop N/A N/A field N/A key N/A N/A