Module Expr Str Import alias Import alias Import alias Import alias ImportFrom alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias identifier ImportFrom alias ImportFrom alias identifier Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr ClassDef bases NameLoad body Pass decorator_list ClassDef bases NameLoad body Pass decorator_list FunctionDef arguments args NameParam defaults body Expr Str If UnaryOpNot AttributeLoad NameLoad attr body Return NameLoad TryExcept body With Call NameLoad AttributeLoad NameLoad attr Str NameStore body Expr Call AttributeLoad NameLoad attr ListLoad Str Str keyword Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Return NameLoad handlers ExceptHandler type AttributeLoad NameLoad attr body Return NameLoad ExceptHandler type NameLoad body Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore BoolOpOr AttributeLoad NameLoad attr AttributeLoad NameLoad attr If CompareIs NameLoad NameLoad body Return NameLoad If Call NameLoad NameLoad body Return NameLoad Return Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Assign NameStore BoolOpOr NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr If UnaryOpNot NameLoad body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr NameLoad If CompareGt Call NameLoad AttributeLoad NameLoad attr NameLoad body Return Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr If CompareIs NameLoad NameLoad body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr Call NameLoad NameLoad SubscriptLoad NameLoad Slice Num If NameLoad body Return Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If CompareNotEq AttributeLoad NameLoad attr Call NameLoad NameLoad body Raise NameLoad If CompareNotEq AttributeLoad NameLoad attr NameLoad body Raise NameLoad Assign NameStore Dict Str Str Str Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad TryExcept body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad NameLoad handlers ExceptHandler type AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad NameLoad AttributeLoad NameLoad attr NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str keyword NameLoad keyword NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore NameLoad If UnaryOpNot NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad If CompareIs NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr For TupleStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr body For TupleStore NameStore NameStore Call NameLoad NameLoad body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad NameLoad NameLoad NameLoad keyword NameLoad keyword NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Break decorator_list FunctionDef arguments args defaults body Return Call AttributeLoad NameLoad attr Call NameLoad Str Str NameLoad decorator_list FunctionDef arguments args NameParam defaults NameLoad body Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr Call NameLoad NameLoad For NameStore NameLoad body TryExcept body With Call NameLoad body Expr Call NameLoad NameLoad keyword NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad decorator_list If CompareEq NameLoad Str body Import alias Assign NameStore CompareIn Str AttributeLoad NameLoad attr Expr Call NameLoad NameLoad Expr Call NameLoad keyword NameLoad keyword NameLoad Expr Call NameLoad keyword NameLoad Expr Call NameLoad NameLoad ImportFrom alias ImportFrom alias ImportFrom alias Import alias ImportFrom alias ClassDef bases NameLoad body FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr Assign AttributeStore NameLoad attr Call NameLoad Assign AttributeStore NameLoad attr AttributeLoad AttributeLoad NameLoad attr attr Assign AttributeStore NameLoad attr Call NameLoad keyword AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad keyword NameLoad For NameStore Call NameLoad Num body Assign NameStore Call AttributeLoad Str attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr keyword Str keyword NameLoad keyword Call NameLoad NameLoad keyword Str decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call NameLoad NameLoad Expr Call NameLoad Call NameLoad AttributeLoad NameLoad attr Num For TupleStore NameStore NameStore Call NameLoad AttributeLoad NameLoad attr body Expr Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad Str attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr BinOpAdd NameLoad Num NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Call NameLoad keyword NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call NameLoad NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad keyword NameLoad Expr Call NameLoad Call NameLoad AttributeLoad NameLoad attr Num decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str keyword NameLoad keyword NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index Str Index Num Assign TupleStore NameStore NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Expr Call NameLoad keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr Expr Call NameLoad keyword NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad keyword NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad keyword NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list
N/A N/A Copy files from previous OSF storage to new OSF storage. Important: settings
must be *exactly* the same as in the production upload service, else files
could be uploaded to the wrong place.
 N/A os N/A hashlib N/A logging N/A subprocess cStringIO StringIO N/A requests modularodm Q framework.transactions.context TokuTransaction website settings website.app init_app website.models Node website.addons.osffiles.model NodeFile website.addons.osfstorage model website.addons.osfstorage utils website.addons.osfstorage errors scripts utils script_utils scripts.osfstorage.utils ensure_osf_files scripts.osfstorage settings scripts_settings N/A logger N/A N/A logging getLogger __name__ N/A N/A N/A script_utils add_file_logger logger __file__ N/A N/A N/A logging basicConfig level N/A logging INFO N/A client N/A N/A scripts_settings STORAGE_CLIENT_CLASS N/A scripts_settings STORAGE_CLIENT_OPTIONS N/A container N/A N/A client create_container N/A scripts_settings STORAGE_CONTAINER_NAME SizeMismatchError N/A Exception N/A N/A N/A HashMismatchError N/A Exception N/A N/A N/A check_node N/A N/A node N/A N/A N/A Check whether git repo for node is intact.
     N/A N/A N/A node files_current N/A N/A True N/A N/A N/A N/A open N/A os devnull w fnull N/A N/A N/A N/A subprocess check_call N/A git log cwd N/A N/A N/A os path join N/A settings UPLOADS_PATH N/A node _id stdout fnull stderr fnull N/A True N/A N/A N/A N/A subprocess CalledProcessError N/A N/A False N/A N/A OSError N/A N/A False N/A get_source_node N/A N/A node N/A N/A N/A Recursively search for source node (`registered_from` or `forked_from`),
    excluding corrupt nodes.
     N/A source N/A N/A node registered_from N/A node forked_from N/A N/A source None N/A N/A None N/A N/A check_node source N/A N/A source N/A N/A get_source_node source N/A migrate_version N/A N/A idx node_file node_settings node dry_run N/A None True N/A N/A Migrate a legacy file version to OSF Storage. If `node` is provided, use
    instead of the `Node` attached to `node_settings`; used when the git repo
    for the current node is missing or corrupt.

    :param int idx: Version index (zero-based)
    :param NodeFile node_file: Legacy file record
    :param OsfStorageNodeSettings node_settings: Node settings
    :param Node node: Optional source node
     N/A node N/A node N/A node_settings owner N/A N/A N/A logger info N/A N/A Migrating version {0} from NodeFile {1} on node {2} format idx N/A node_file _id N/A node _id N/A N/A dry_run N/A N/A record N/A N/A N/A model OsfStorageFileRecord get_or_create N/A node_file path node_settings N/A N/A N/A len N/A record versions idx N/A N/A N/A content N/A N/A N/A scripts_settings SPECIAL_CASES get N/A N/A node _id N/A node_file _id N/A N/A content None N/A N/A N/A content _ N/A N/A node read_file_object node_file N/A N/A N/A logger info N/A N/A Loaded content with length {0}: {1}... format N/A len content N/A content N/A 10 N/A dry_run N/A N/A N/A md5 N/A N/A N/A N/A hashlib md5 content hexdigest N/A file_pointer N/A StringIO content N/A hash_str N/A N/A N/A N/A scripts_settings UPLOAD_PRIMARY_HASH content hexdigest N/A obj N/A N/A container get_or_upload_file file_pointer hash_str N/A N/A N/A obj size N/A len content N/A N/A SizeMismatchError N/A N/A N/A obj md5 md5 N/A N/A HashMismatchError N/A metadata N/A size content_type date_modified md5 N/A obj size N/A obj content_type N/A N/A N/A obj date_modified isoformat md5 N/A N/A N/A N/A N/A record create_pending_version N/A node_file uploader N/A N/A {}-{} format hash_str idx N/A N/A N/A N/A errors OsfStorageError N/A N/A latest_version N/A N/A record get_version required True N/A N/A N/A record remove_version latest_version N/A record N/A N/A OsfStorageFileRecord get_or_create N/A node_file path node_settings N/A N/A N/A record create_pending_version N/A node_file uploader N/A N/A {}-{} format hash_str idx N/A N/A N/A record resolve_pending_version N/A N/A {}-{} format hash_str idx N/A obj location metadata log False N/A migrate_node N/A N/A node dry_run N/A True N/A N/A Migrate legacy files for a node. If the git repo for the node is corrupt,
    attempt to use its source node (registration or fork) instead.
     N/A N/A N/A logger info N/A N/A Migrating node {0} format N/A node _id N/A node_settings N/A N/A node get_or_add_addon osfstorage auth None log False N/A repo_intact N/A check_node node N/A source_node None N/A N/A repo_intact N/A N/A N/A N/A logger warn N/A N/A Original node {0} is corrupt; attempting to recover format N/A node _id N/A source_node N/A get_source_node node N/A N/A source_node None N/A N/A N/A N/A logger error N/A N/A Could not identify source node for recovery on node {0} format N/A node _id N/A N/A path versions N/A N/A N/A node files_versions iteritems N/A N/A N/A idx version N/A enumerate versions N/A N/A N/A N/A node_file N/A N/A NodeFile load version N/A N/A migrate_version idx node_file node_settings node source_node dry_run dry_run N/A N/A N/A Exception N/A error N/A N/A N/A N/A logger error N/A N/A Could not migrate object {0} on node {1} format version N/A node _id N/A N/A N/A logger exception error N/A N/A get_nodes N/A N/A N/A N/A N/A N/A N/A Node find N/A Q files_versions ne None N/A main N/A N/A dry_run N/A True N/A N/A nodes N/A get_nodes N/A N/A N/A logger info N/A N/A Migrating files on {0} `Node` records format N/A len nodes N/A node nodes N/A N/A N/A N/A N/A TokuTransaction N/A N/A N/A migrate_node node dry_run dry_run N/A N/A N/A Exception N/A error N/A N/A N/A N/A logger error N/A N/A Could not migrate node {0} format N/A node _id N/A N/A N/A logger exception error N/A N/A N/A __name__ __main__ N/A N/A sys N/A dry_run N/A dry N/A sys argv N/A N/A ensure_osf_files settings N/A N/A init_app set_backends True routes False N/A N/A main dry_run dry_run N/A N/A ensure_osf_files settings nose.tools * tests.base OsfTestCase tests.factories ProjectFactory N/A shutil framework.auth Auth TestMigrateFiles N/A OsfTestCase N/A setUp N/A N/A self N/A N/A N/A N/A N/A N/A super TestMigrateFiles self setUp N/A N/A self project N/A ProjectFactory N/A N/A self user N/A N/A self project creator N/A N/A self auth_obj N/A Auth user N/A self user N/A N/A N/A N/A self project delete_addon osfstorage auth None _force True N/A idx N/A range 5 N/A N/A content N/A N/A i want {0} pizzas format idx N/A N/A N/A N/A self project add_file auth N/A self auth_obj file_name pizza.md content content size N/A len content content_type text/markdown N/A check_record N/A N/A self record N/A N/A N/A N/A assert_true record N/A N/A assert_equal N/A len N/A record versions 5 N/A N/A idx version N/A enumerate N/A record versions N/A N/A N/A assert_false N/A version pending N/A expected N/A N/A i want {0} pizzas format idx N/A download_url N/A N/A utils get_waterbutler_download_url N/A idx 1 version record N/A resp N/A N/A requests get download_url N/A N/A assert_equal expected N/A resp content N/A test_migrate N/A N/A self N/A N/A N/A N/A main dry_run False N/A node_settings N/A N/A N/A self project get_addon osfstorage N/A N/A assert_true node_settings N/A record N/A N/A N/A model OsfStorageFileRecord find_by_path pizza.md node_settings N/A N/A N/A self check_record record N/A N/A main dry_run False N/A N/A assert_equal N/A len N/A record versions 5 N/A test_migrate_incomplete N/A N/A self N/A N/A N/A node_settings N/A N/A N/A self project get_or_add_addon osfstorage auth None log False N/A record N/A N/A N/A model OsfStorageFileRecord get_or_create pizza.md node_settings N/A node_file N/A N/A NodeFile load N/A N/A N/A N/A self project files_versions N/A pizza_md N/A 0 N/A N/A content _ N/A N/A N/A self project read_file_object node_file N/A file_pointer N/A StringIO content N/A hash_str N/A N/A N/A N/A scripts_settings UPLOAD_PRIMARY_HASH content hexdigest N/A N/A N/A record create_pending_version N/A node_file uploader hash_str N/A N/A main dry_run False N/A test_migrate_fork N/A N/A self N/A N/A N/A fork N/A N/A N/A self project fork_node auth N/A self auth_obj N/A N/A main dry_run False N/A node_settings N/A N/A N/A self project get_addon osfstorage N/A record N/A N/A N/A model OsfStorageFileRecord find_by_path pizza.md node_settings N/A N/A N/A self check_record record N/A fork_node_settings N/A N/A fork get_addon osfstorage N/A fork_record N/A N/A N/A model OsfStorageFileRecord find_by_path pizza.md fork_node_settings N/A N/A N/A self check_record fork_record N/A test_migrate_corrupt_fork_repo_deleted N/A N/A self N/A N/A N/A fork N/A N/A N/A self project fork_node auth N/A self auth_obj N/A fork_repo N/A N/A N/A os path join N/A settings UPLOADS_PATH N/A fork _id N/A N/A N/A shutil rmtree fork_repo N/A N/A main dry_run False N/A node_settings N/A N/A N/A self project get_addon osfstorage N/A record N/A N/A N/A model OsfStorageFileRecord find_by_path pizza.md node_settings N/A N/A N/A self check_record record N/A fork_node_settings N/A N/A fork get_addon osfstorage N/A fork_record N/A N/A N/A model OsfStorageFileRecord find_by_path pizza.md fork_node_settings N/A N/A N/A self check_record fork_record N/A test_migrate_corrupt_fork_git_dir_deleted N/A N/A self N/A N/A N/A fork N/A N/A N/A self project fork_node auth N/A self auth_obj N/A fork_git_dir N/A N/A N/A os path join N/A settings UPLOADS_PATH N/A fork _id .git N/A N/A N/A shutil rmtree fork_git_dir N/A N/A main dry_run False N/A node_settings N/A N/A N/A self project get_addon osfstorage N/A record N/A N/A N/A model OsfStorageFileRecord find_by_path pizza.md node_settings N/A N/A N/A self check_record record N/A fork_node_settings N/A N/A fork get_addon osfstorage N/A fork_record N/A N/A N/A model OsfStorageFileRecord find_by_path pizza.md fork_node_settings N/A N/A N/A self check_record fork_record N/A N/A