Module Import alias Import alias Import alias identifier Import alias ImportFrom alias ImportFrom alias ImportFrom alias identifier ImportFrom alias ImportFrom alias ImportFrom alias alias alias alias alias alias alias alias ImportFrom alias ImportFrom alias ImportFrom alias identifier ImportFrom alias identifier ImportFrom alias alias ImportFrom alias ImportFrom alias alias FunctionDef arguments args defaults body TryExcept body Expr Call NameLoad NameLoad Expr Call NameLoad NameLoad handlers ExceptHandler type NameLoad body Pass decorator_list Assign NameStore Call NameLoad NameLoad ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults vararg kwarg body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam defaults vararg kwarg body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam NameParam defaults vararg kwarg body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Call NameLoad Str Str NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list decorator_list ClassDef bases AttributeLoad NameLoad attr body Assign NameStore NameLoad Assign NameStore Num FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str If UnaryOpNot NameLoad body Raise Call NameLoad Dict Str Str Str Str NameLoad NameLoad NameLoad NameLoad If CompareEq AttributeLoad NameLoad attr NameLoad body Return Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore ListLoad If Call NameLoad NameLoad NameLoad body Assign AttributeStore NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr orelse If Call NameLoad NameLoad NameLoad body Assign AttributeStore NameLoad attr NameLoad Assign NameStore ListComp NameLoad comprehension NameStore Call AttributeLoad AttributeLoad NameLoad attr attr CompareIsNot NameLoad NameLoad orelse If Call NameLoad NameLoad NameLoad body Assign AttributeStore NameLoad attr NameLoad Assign NameStore Dict Str Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr orelse Assign AttributeStore NameLoad attr NameLoad Assign NameStore IfExp NameLoad ListLoad NameLoad ListLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword NameLoad decorator_list decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore NameLoad Assign NameStore NameLoad If CompareIn Str NameLoad body Assign NameStore Str Assign NameStore IfExp CompareEq SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Str Str Str Expr Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad TryExcept body Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad keyword ListLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Raise Assign NameStore Call NameLoad AttributeLoad NameLoad attr NameLoad keyword ListLoad Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Return NameLoad decorator_list Call AttributeLoad NameLoad attr keyword NameLoad keyword Str Call NameLoad Str FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Expr Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword Call AttributeLoad NameLoad attr NameLoad If CompareNotIn AttributeLoad NameLoad attr TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Raise Call NameLoad AttributeLoad NameLoad attr decorator_list Call AttributeLoad NameLoad attr keyword NameLoad keyword Str Call NameLoad Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam defaults NameLoad body Assign NameStore Dict Str Str Str Dict Str Str Str Str NameLoad AttributeLoad NameLoad attr NameLoad Str Dict Str Str Str Str NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Call AttributeLoad NameLoad attr Str Str If NameLoad body Assign SubscriptStore SubscriptLoad NameLoad Index Str Index Str NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore NameLoad If CompareIn Str NameLoad body Assign NameStore Str Expr Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore BinOpAdd AttributeLoad NameLoad attr Str If CompareEq NameLoad Str body Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Call AttributeLoad Str attr NameLoad NameLoad keyword Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Call AttributeLoad Str attr NameLoad NameLoad keyword Str Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad orelse Assign NameStore Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad decorator_list Call AttributeLoad NameLoad attr keyword NameLoad keyword Str Call NameLoad Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr If UnaryOpNot Call NameLoad NameLoad NameLoad body Assign NameStore ListLoad NameLoad Assign NameStore Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad If BoolOpAnd CompareNotIn NameLoad AttributeLoad AttributeLoad NameLoad attr attr CompareGt AttributeLoad NameLoad attr AttributeLoad NameLoad attr body Raise Call NameLoad keyword NameLoad orelse If UnaryOpNot AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr For NameStore AttributeLoad NameLoad attr body If UnaryOpNot AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad orelse Expr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list Call AttributeLoad NameLoad attr keyword NameLoad keyword Str Call NameLoad Str FunctionDef arguments args NameParam defaults body Expr Str Expr Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return Call AttributeLoad NameLoad attr ListLoad Call AttributeLoad NameLoad attr GeneratorExp Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword NameLoad comprehension NameStore AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad For NameStore AttributeLoad NameLoad attr body If AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If UnaryOpNot AttributeLoad NameLoad attr body Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr keyword NameLoad decorator_list Call AttributeLoad NameLoad attr keyword NameLoad keyword Str Call NameLoad Str
N/A N/A requests N/A json N/A httplib http N/A celery celery.utils.log get_task_logger modularodm Q framework.celery_tasks app celery_app framework.celery_tasks.utils logged framework.exceptions HTTPError website.archiver ARCHIVER_SUCCESS ARCHIVER_FAILURE ARCHIVER_SIZE_EXCEEDED ARCHIVER_NETWORK_ERROR ARCHIVER_FILE_NOT_FOUND ARCHIVER_UNCAUGHT_ERROR NO_ARCHIVE_LIMIT AggregateStatResult website.archiver utils website.archiver.model ArchiveJob website.archiver signals archiver_signals website.project signals project_signals website.project.model Node DraftRegistration website settings website.app init_addons do_set_backends create_app_context N/A N/A N/A N/A N/A N/A N/A N/A init_addons settings N/A N/A do_set_backends settings N/A N/A N/A AssertionError N/A N/A N/A N/A logger N/A get_task_logger __name__ ArchiverSizeExceeded N/A Exception N/A __init__ N/A N/A self result N/A args kwargs N/A N/A N/A N/A N/A super ArchiverSizeExceeded self __init__ args kwargs N/A N/A self result result N/A N/A ArchiverStateError N/A Exception N/A __init__ N/A N/A self info N/A args kwargs N/A N/A N/A N/A N/A super ArchiverStateError self __init__ args kwargs N/A N/A self info info N/A N/A ArchivedFileNotFound N/A Exception N/A __init__ N/A N/A self registration missing_files N/A args kwargs N/A N/A N/A N/A N/A super ArchivedFileNotFound self __init__ args kwargs N/A N/A self draft_registration N/A N/A DraftRegistration find_one N/A Q registered_node eq registration N/A N/A self missing_files missing_files N/A N/A ArchiverTask N/A N/A celery Task N/A N/A abstract True N/A max_retries 0 on_failure N/A N/A self exc task_id args kwargs einfo N/A N/A N/A job N/A N/A ArchiveJob load N/A N/A kwargs get job_pk N/A N/A job N/A N/A N/A ArchiverStateError N/A exception args kwargs einfo exc args kwargs einfo N/A N/A N/A job status ARCHIVER_FAILURE N/A N/A N/A N/A src dst user N/A N/A job info N/A errors N/A N/A N/A isinstance exc ArchiverSizeExceeded N/A N/A N/A dst archive_status ARCHIVER_SIZE_EXCEEDED N/A errors N/A exc result N/A N/A N/A isinstance exc HTTPError N/A N/A N/A dst archive_status ARCHIVER_NETWORK_ERROR N/A errors N/A each N/A each N/A N/A N/A dst archive_job target_info N/A each None N/A N/A N/A isinstance exc ArchivedFileNotFound N/A N/A N/A dst archive_status ARCHIVER_FILE_NOT_FOUND N/A errors N/A missing_files draft N/A exc missing_files N/A exc draft_registration N/A N/A N/A dst archive_status ARCHIVER_UNCAUGHT_ERROR N/A errors N/A einfo N/A einfo N/A N/A N/A N/A dst save N/A N/A N/A N/A archiver_signals archive_fail send dst errors errors N/A N/A stat_addon N/A N/A addon_short_name job_pk N/A N/A N/A Collect metadata about the file tree of a given addon

    :param addon_short_name: AddonConfig.short_name of the addon to be examined
    :param job_pk: primary key of archive_job
    :return: AggregateStatResult containing file tree metadata
     N/A addon_name addon_short_name N/A version None N/A N/A dataverse addon_short_name N/A N/A addon_name dataverse N/A version N/A N/A N/A N/A N/A addon_short_name split - N/A -1 draft latest latest-published N/A N/A create_app_context N/A job N/A N/A ArchiveJob load job_pk N/A N/A src dst user N/A N/A job info N/A src_addon N/A N/A src get_addon addon_name N/A N/A N/A file_tree N/A N/A src_addon _get_file_tree user user version version N/A N/A N/A HTTPError N/A e N/A N/A N/A N/A N/A dst archive_job update_target addon_short_name ARCHIVER_NETWORK_ERROR errors N/A N/A N/A e data N/A error N/A N/A result N/A AggregateStatResult N/A src_addon _id addon_short_name targets N/A N/A N/A utils aggregate_file_tree_metadata addon_short_name file_tree user N/A result N/A N/A N/A celery_app task base ArchiverTask name archiver.stat_addon N/A logged stat_addon make_copy_request N/A N/A job_pk url data N/A N/A N/A Make the copy request to the WaterBulter API and handle
    successful and failed responses

    :param job_pk: primary key of ArchiveJob
    :param url: URL to send request to
    :param data: <dict> of setting to send in POST to WaterBulter API
    :return: None
     N/A N/A create_app_context N/A job N/A N/A ArchiveJob load job_pk N/A N/A src dst user N/A N/A job info N/A provider N/A N/A data N/A source N/A provider N/A N/A N/A logger info N/A N/A Sending copy request for addon: {0} on node: {1} format provider N/A dst _id N/A res N/A N/A requests post url data N/A N/A json dumps data N/A N/A N/A res status_code N/A N/A http OK N/A http CREATED N/A http ACCEPTED N/A N/A N/A HTTPError N/A res status_code N/A N/A N/A celery_app task base ArchiverTask name archiver.make_copy_request N/A logged make_copy_request make_waterbutler_payload N/A N/A src dst addon_short_name rename cookie revision N/A None N/A N/A ret N/A source destination rename N/A cookie nid provider path cookie N/A src _id addon_short_name / N/A cookie nid provider path cookie N/A dst _id N/A settings ARCHIVE_PROVIDER / N/A N/A rename replace / - N/A revision N/A N/A N/A N/A ret N/A source N/A revision revision N/A ret N/A archive_addon N/A N/A addon_short_name job_pk stat_result N/A N/A N/A Archive the contents of an addon by making a copy request to the
    WaterBulter API

    :param addon_short_name: AddonConfig.short_name of the addon to be archived
    :param job_pk: primary key of ArchiveJob
    :return: None
     N/A addon_name addon_short_name N/A N/A dataverse addon_short_name N/A N/A addon_name dataverse N/A N/A create_app_context N/A job N/A N/A ArchiveJob load job_pk N/A N/A src dst user N/A N/A job info N/A N/A N/A logger info N/A N/A Archiving addon: {0} on node: {1} format addon_short_name N/A src _id N/A src_provider N/A N/A src get_addon addon_name N/A folder_name N/A src_provider archive_folder_name N/A cookie N/A N/A user get_or_create_cookie N/A copy_url N/A N/A settings WATERBUTLER_URL /ops/copy N/A N/A addon_name dataverse N/A N/A data N/A make_waterbutler_payload src dst addon_name N/A N/A {0} (published) format folder_name cookie revision latest-published N/A N/A N/A make_copy_request delay job_pk job_pk url copy_url data data N/A data N/A make_waterbutler_payload src dst addon_name N/A N/A {0} (draft) format folder_name cookie revision latest N/A N/A N/A make_copy_request delay job_pk job_pk url copy_url data data N/A N/A data N/A make_waterbutler_payload src dst addon_name folder_name cookie N/A N/A N/A make_copy_request delay job_pk job_pk url copy_url data data N/A N/A N/A celery_app task base ArchiverTask name archiver.archive_addon N/A logged archive_addon archive_node N/A N/A stat_results job_pk N/A N/A N/A First use the results of #stat_node to check disk usage of the
    initiated registration, then either fail the registration or
    create a celery.group group of subtasks to archive addons

    :param results: results from the #stat_addon subtasks spawned in #stat_node
    :param job_pk: primary key of ArchiveJob
    :return: None
     N/A N/A create_app_context N/A job N/A N/A ArchiveJob load job_pk N/A N/A src dst user N/A N/A job info N/A N/A N/A logger info N/A N/A Archiving node: {0} format N/A src _id N/A N/A N/A isinstance stat_results list N/A N/A stat_results N/A stat_results N/A stat_result N/A AggregateStatResult N/A dst _id N/A dst title targets stat_results N/A N/A N/A NO_ARCHIVE_LIMIT N/A N/A job initiator system_tags N/A N/A stat_result disk_usage N/A settings MAX_ARCHIVE_SIZE N/A N/A N/A ArchiverSizeExceeded result stat_result N/A N/A N/A N/A stat_result targets N/A N/A N/A job status ARCHIVER_SUCCESS N/A N/A N/A job save N/A result N/A stat_result targets N/A N/A N/A N/A result num_files N/A N/A N/A N/A job update_target N/A result target_name ARCHIVER_SUCCESS N/A N/A N/A N/A archive_addon delay addon_short_name N/A result target_name job_pk job_pk stat_result result N/A N/A N/A N/A project_signals archive_callback send dst N/A N/A N/A celery_app task base ArchiverTask name archiver.archive_node N/A logged archive_node archive N/A N/A job_pk N/A N/A N/A Starts a celery.chord that runs stat_addon for each
    complete addon attached to the Node, then runs
    #archive_node with the result

    :param job_pk: primary key of ArchiveJob
    :return: None
     N/A N/A create_app_context N/A job N/A N/A ArchiveJob load job_pk N/A N/A src dst user N/A N/A job info N/A logger N/A get_task_logger __name__ N/A N/A N/A logger info N/A N/A Received archive task for Node: {0} into Node: {1} format N/A src _id N/A dst _id N/A N/A N/A celery chain N/A N/A N/A celery group N/A N/A N/A stat_addon si addon_short_name N/A target name job_pk job_pk N/A target N/A job target_addons N/A N/A archive_node s job_pk job_pk N/A archive_success N/A N/A dst_pk job_pk N/A N/A N/A Archiver's final callback. For the time being the use case for this task
    is to rewrite references to files selected in a registration schema (the Prereg
    Challenge being the first to expose this feature). The created references point
    to files on the registered_from Node (needed for previewing schema data), and
    must be re-associated with the corresponding files in the newly created registration.

    :param str dst_pk: primary key of registration Node

    note:: At first glance this task makes redundant calls to utils.get_file_map (which
    returns a generator yielding  (<sha256>, <file_metadata>) pairs) on the dst Node. Two
    notes about utils.get_file_map: 1) this function memoizes previous results to reduce
    overhead and 2) this function returns a generator that lazily fetches the file metadata
    of child Nodes (it is possible for a selected file to belong to a child Node) using a
    non-recursive DFS. Combined this allows for a relatively effient implementation with
    seemingly redundant calls.
     N/A N/A create_app_context N/A dst N/A N/A Node load dst_pk N/A schema N/A dst registered_schema N/A N/A N/A schema has_files N/A N/A N/A N/A utils migrate_file_metadata dst schema N/A job N/A N/A ArchiveJob load job_pk N/A N/A N/A job sent N/A N/A N/A job sent True N/A N/A N/A job save N/A N/A N/A N/A dst sanction ask N/A N/A dst get_active_contributors_recursive unique_users True N/A N/A N/A celery_app task base ArchiverTask name archiver.archive_success N/A logged archive_success