Module Expr Str Import alias Import alias Import alias Import alias ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call NameLoad AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad AttributeLoad NameLoad attr attr Expr Call NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Set Str Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr If CompareIn NameLoad NameLoad body Assign NameStore Str Assign NameStore NameLoad Return TupleLoad NameLoad NameLoad NameLoad decorator_list NameLoad FunctionDef arguments args NameParam defaults body Expr Str Return AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Return Call NameLoad IfExp CompareIs NameLoad NameLoad AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore ListComp BinOpAdd Str Call NameLoad NameLoad comprehension NameStore NameLoad Return Call NameLoad AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Return Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str Assign NameStore SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index NameLoad Assign NameStore Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return SubscriptLoad AttributeLoad NameLoad attr Index Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad Call AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad Call AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Dict For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body For TupleStore TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr body For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Assign NameStore TupleLoad NameLoad NameLoad SubscriptLoad NameLoad Index Str Expr Call AttributeLoad NameLoad attr NameLoad Dict Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr NameLoad Dict Assign SubscriptStore SubscriptLoad SubscriptLoad NameLoad Index NameLoad Index NameLoad Index NameLoad NameLoad Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr If CompareIs NameLoad NameLoad body Raise Call NameLoad BinOpMod Str Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Call AttributeLoad NameLoad attr If CompareIs NameLoad NameLoad body Assign NameStore BinOpMod Str TupleLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Raise Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr If CompareIs NameLoad NameLoad body Raise Call NameLoad BinOpMod Str Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str TryExcept body Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad body Raise Call NameLoad Str For NameStore Call AttributeLoad NameLoad attr body For NameStore Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str decorator_list decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad body Expr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr If CompareIsNot AttributeLoad NameLoad attr NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str If CompareIs NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr orelse Expr Call AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr If CompareIsNot AttributeLoad NameLoad attr NameLoad body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Assert CompareIs NameLoad NameLoad Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str For NameStore AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore SetComp Call AttributeLoad NameLoad attr comprehension NameStore AttributeLoad NameLoad attr For NameStore AttributeLoad NameLoad attr body If CompareNotIn NameLoad NameLoad body If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str For NameStore AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If CompareNotIn NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str Assign NameStore Num Assign NameStore Num FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call NameLoad NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str For NameStore Call NameLoad BinOpAdd AttributeLoad NameLoad attr Num body TryExcept body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Break handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr orelse Raise Call NameLoad BinOpAdd BinOpMod Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str For NameStore AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str If NameLoad body If Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad NameLoad Index Num body TryExcept body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Slice Num handlers ExceptHandler body Expr Call AttributeLoad AttributeLoad NameLoad attr attr SubscriptLoad NameLoad Index Num Raise orelse Raise Call NameLoad decorator_list decorator_list ClassDef bases NameLoad body Expr Str Pass decorator_list
N/A N/A 
Collection: database interface class.

These classes provide an interface between the database and the top-level
ingest algorithm (AbstractIngester and its subclasses). They also provide
the implementation of the database and tile store side of the ingest
process. They are expected to be independent of the structure of any
particular dataset, but will change if the database schema or tile store
format changes.
 N/A logging N/A os N/A time N/A shutil agdc.cube_util DatasetError create_directory tile_contents TileContents acquisition_record AcquisitionRecord ingest_db_wrapper IngestDBWrapper N/A LOGGER N/A N/A logging getLogger __name__ N/A N/A N/A LOGGER setLevel N/A logging INFO Collection N/A object N/A N/A Collection database interface class. __init__ N/A N/A self datacube N/A N/A N/A Initialise the collection object. N/A N/A self datacube datacube N/A N/A self db N/A IngestDBWrapper N/A datacube db_connection N/A N/A self new_bands N/A N/A self __reindex_bands N/A datacube bands N/A N/A self transaction_stack N/A N/A N/A self temp_tile_directory N/A N/A N/A os path join N/A N/A self datacube tile_root ingest_temp N/A N/A self datacube process_id N/A N/A create_directory N/A self temp_tile_directory N/A cleanup N/A N/A self N/A N/A N/A Do end-of-process cleanup.

        Deletes the process-specific temporary dirctory. Does not
        close the database connection (at present), because the datacube
        object has a destructor which does that.
         N/A N/A N/A shutil rmtree N/A self temp_tile_directory ignore_errors True N/A get_dataset_key N/A N/A dataset N/A N/A N/A Return the dataset key for use with the new_bands dictionary.

        This is a tuple (satellite_tag, sensor_name, processing_level) except
        that for derived datasets (currently PQA and FC) the satellite_tag is
        replaced with 'DERIVED' and the processing_level is used as the
        sensor_name. So the tuple looks like:
        ('DERIVED', processing_level, processing_level).
         N/A derived_levels N/A PQA FC N/A satellite N/A N/A dataset get_satellite_tag N/A sensor N/A N/A dataset get_sensor_name N/A level N/A N/A dataset get_processing_level N/A N/A level derived_levels N/A N/A satellite DERIVED N/A sensor level N/A N/A satellite sensor level N/A staticmethod get_temp_tile_directory N/A N/A self N/A N/A N/A Return a path to a directory for temporary tile related files. N/A N/A self temp_tile_directory N/A check_metadata N/A N/A self dataset N/A N/A N/A Check that the satellite, sensor, and bands are in the database.

        Checks that the dataset is of a kind that the database knows about
        (by checking basic metadata), and the bands that the database expects
        are present. Raises a DatasetError if the checks fail.
         N/A N/A N/A self __check_satellite_and_sensor dataset N/A N/A N/A self __check_processing_level dataset N/A N/A N/A self __check_bands dataset N/A transaction N/A N/A self db N/A None N/A N/A Returns a Transaction context manager object.

        This is for use in a 'with' statement. It uses the Collection's
        database collection if one is not provided.
         N/A N/A Transaction N/A N/A db None N/A self db db N/A self transaction_stack N/A lock_datasets N/A N/A self dataset_list N/A N/A N/A Returns a Lock context manager object.

        dataset_list is a list of dataset ids for the datasets to be
        locked.

        This is for use in a 'with' statement. It uses the Collection's
        datacube object to manage the individual locks.
         N/A lock_list N/A N/A Dataset- N/A str dataset_id N/A dataset_id dataset_list N/A N/A Lock N/A self datacube lock_list N/A create_acquisition_record N/A N/A self dataset N/A N/A N/A Factory method to create an instance of the AcquisitonRecord class.

        This method creates a corresponding record in the database if one
        does not already exist.
         N/A N/A AcquisitionRecord self dataset N/A create_tile_contents N/A N/A self tile_type_id tile_footprint band_stack N/A N/A N/A Factory method to create an instance of the TileContents class.

        The tile_type_dict contains the information required for
        resampling extents and resolution.
         N/A tile_type_info N/A N/A N/A self datacube tile_type_dict N/A tile_type_id N/A tile_contents N/A TileContents N/A N/A self datacube tile_root tile_type_info tile_footprint band_stack N/A tile_contents N/A current_transaction N/A N/A self N/A N/A N/A Returns the current transaction. N/A N/A N/A self transaction_stack N/A -1 N/A mark_tile_for_removal N/A N/A self tile_pathname N/A N/A N/A Mark a tile file for removal on transaction commit. N/A N/A N/A N/A N/A self current_transaction mark_tile_for_removal tile_pathname N/A mark_tile_for_creation N/A N/A self tile_contents N/A N/A N/A Mark a tile file for creation on transaction commit. N/A N/A N/A N/A N/A self current_transaction mark_tile_for_creation tile_contents N/A __reindex_bands N/A N/A bands N/A N/A N/A Reindex the datacube.bands nested dict structure.

        This method returns the new nested dict which is indexed by:
            new_bands[dataset_key][tile_type][file_number]
        where dataset_key is a tuple:
            (satellite_tag, sensor_name, processing_level).

        The original indexing is
            bands[tile_type][satellite_sensor][file_number]
        where satellite_sensor is a tuple:
            (satellite_tag, sensor_name)

        Note that satellite_tag and sensor_name are replaced by 'DERIVED' and
        the processing_level for PQA and FC datasets. This needs to be taken
        into account when constructing a dataset_key.
         N/A new_bands N/A N/A N/A tile_type band_dict N/A N/A bands items N/A N/A N/A N/A satellite sensor sensor_dict N/A N/A band_dict items N/A N/A N/A file_number band_info N/A N/A sensor_dict items N/A N/A dataset_key N/A satellite sensor N/A band_info N/A level_name N/A N/A N/A new_bands setdefault dataset_key N/A N/A N/A N/A N/A new_bands N/A dataset_key setdefault tile_type N/A N/A N/A N/A N/A new_bands N/A dataset_key N/A tile_type N/A file_number band_info N/A new_bands N/A staticmethod __check_satellite_and_sensor N/A N/A self dataset N/A N/A N/A Check that the dataset's satellite and sensor are in the database.

        Raises a DatasetError if they are not.
         N/A satellite_id N/A N/A N/A self db get_satellite_id N/A N/A dataset get_satellite_tag N/A N/A satellite_id None N/A N/A N/A DatasetError N/A Unknown satellite tag: '%s' N/A N/A dataset get_satellite_tag N/A sensor_id N/A N/A N/A self db get_sensor_id satellite_id N/A N/A dataset get_sensor_name N/A N/A sensor_id None N/A N/A msg N/A Unknown satellite and sensor pair: '%s', '%s' N/A N/A N/A dataset get_satellite_tag N/A N/A dataset get_sensor_name N/A N/A DatasetError msg N/A __check_processing_level N/A N/A self dataset N/A N/A N/A Check that the dataset's processing_level is in the database.

        Raises a DatasetError if it is not.
         N/A level_id N/A N/A N/A self db get_level_id N/A N/A dataset get_processing_level N/A N/A level_id None N/A N/A N/A DatasetError N/A Unknown processing level: '%s' N/A N/A dataset get_processing_level N/A __check_bands N/A N/A self dataset N/A N/A N/A Check that the dataset has the expected bands.

        Raises a DatasetError if any band expected for this dataset (according
        to the database) is missing.
         N/A N/A N/A dataset_bands N/A N/A self new_bands N/A N/A N/A self get_dataset_key dataset N/A N/A N/A KeyError N/A N/A N/A DatasetError No tile types for this dataset. N/A tile_type_bands N/A N/A dataset_bands values N/A N/A band_info N/A N/A tile_type_bands values N/A N/A N/A N/A dataset find_band_file N/A band_info N/A file_pattern N/A N/A Transaction N/A object N/A N/A Context manager class for a transaction involving tiles.

    This is used in a 'with' statement to wrap a transaction.
    It handles the commit or roll back of the transaction and
    the associated file operations to create and remove tile files
    in coordination with the transaction.
     __init__ N/A N/A self db tr_stack N/A None N/A N/A Initialise the transaction.

        db is the database connection to use.
        tr_stack is a stack of transactions. If not None, the last item
            on the tr_stack should be the current transaction.
        tile_remove_list is the list of tile files to remove on commit.
        tile_create_list is the list of tile contents to create on commit
        (or cleanup on roll back).
        previous_commit_mode holds the previous state of the connection
        so that it can be restored when the transaction is finished.

        Note that tile_create_list is a list of TileContent objects,
        while tile_remove_list is a list of pathnames.
         N/A N/A self db db N/A N/A self tr_stack tr_stack N/A N/A self tile_remove_list None N/A N/A self tile_create_list None N/A N/A self previous_commit_mode None N/A __enter__ N/A N/A self N/A N/A N/A Auto-called on transaction (with statement) entry.

        Clears the tile lists and sets the commit mode (saving the old one).
        Returns 'self' so that the other methods are available via an
        'as' clause.

        Note that tile_create_list is a list of TileContent objects,
        while tile_remove_list is a list of pathnames.
         N/A N/A self tile_remove_list N/A N/A N/A self tile_create_list N/A N/A N/A self previous_commit_mode N/A N/A N/A self db turn_off_autocommit N/A N/A N/A self tr_stack None N/A N/A N/A N/A N/A self tr_stack append self N/A self N/A __exit__ N/A N/A self exc_type exc_value exc_traceback N/A N/A N/A Auto called on transaction (with statement) exit.

        Commits the transaction unless there has been an exception,
        in which case it rolls back the transaction. Restores the
        old commit mode to the database connection.

        This implicitly returns None which causes any exception
        to be re-raised.
         N/A N/A exc_type None N/A N/A N/A N/A self __commit N/A N/A N/A N/A self __rollback N/A N/A self tile_remove_list None N/A N/A self tile_create_list None N/A N/A N/A N/A self db restore_commit_mode N/A self previous_commit_mode N/A N/A N/A self tr_stack None N/A N/A tr N/A N/A N/A self tr_stack pop N/A N/A tr self Unexpected value on transaction stack. N/A __commit N/A N/A self N/A N/A N/A Commit the transaction while handling tile files. N/A tile_contents N/A self tile_create_list N/A N/A N/A N/A tile_contents make_permanent N/A N/A N/A N/A self db commit N/A tile_create_set N/A N/A N/A t get_output_path N/A t N/A self tile_create_list N/A tile_pathname N/A self tile_remove_list N/A N/A N/A tile_pathname tile_create_set N/A N/A N/A N/A N/A os path isfile tile_pathname N/A N/A N/A N/A os remove tile_pathname N/A __rollback N/A N/A self N/A N/A N/A Roll back the transaction while handling tile files. N/A tile_contents N/A self tile_create_list N/A N/A N/A N/A tile_contents remove N/A N/A N/A N/A self db rollback N/A mark_tile_for_removal N/A N/A self tile_pathname N/A N/A N/A Mark a tile file for removal on transaction commit.

        These tiles will be deleted if the transaction is commited,
        but not if it is rolled back.
         N/A N/A tile_pathname N/A self tile_remove_list N/A N/A N/A N/A N/A self tile_remove_list append tile_pathname N/A mark_tile_for_creation N/A N/A self tile_contents N/A N/A N/A Mark a tile file for creation on transaction commit.

        These tiles will be created (moved to their permenant
        location) if the transaction is commited. If the transaction
        is rolled back the associated temprorary tile files will be
        removed.

        tile_contents should be a TileContents object (or at least
        implement the interface).
         N/A N/A N/A N/A self tile_create_list append tile_contents N/A N/A Lock N/A object N/A N/A Context manager class for locking a list of objects.

    This is used in a 'with' statement to wrap code which needs the
    locks. It handles acquiring and releasing the locks as well as
    waiting and retries if the locks cannot be acquired.

    Not that this will not work for nested locks/with statements in the
    same process that attempt to lock the same object, because the
    locking mechanism  does not count the number of times an object has
    been locked.
     N/A DEFAULT_WAIT 10 N/A DEFAULT_RETRIES 6 __init__ N/A N/A self datacube lock_list wait retries N/A DEFAULT_WAIT DEFAULT_RETRIES N/A N/A Initialise the lock object.

        Positional Arguments:
            datacube: The datacube object which manages the individual locks.
            lock_list: The list of objects to lock. This is a list of
                strings - each string should unambiguously identify the object
                that is being locked.

        Keyword Arguments:
            wait: The amount of time to wait, in seconds, before again trying
                to acquire the locks.
            retries: The maximum number of attempts before giving up and
                raising an exception.
         N/A N/A self datacube datacube N/A N/A self lock_list N/A sorted lock_list N/A N/A self wait wait N/A N/A self retries retries N/A __enter__ N/A N/A self N/A N/A N/A Auto-called on 'with' statement entry.

        This acquires the locks or raises a LockError if it cannot
        do so (after the maximum number of tries). Note that LockError
        is a subclass of DatasetError, so it will cause a dataset skip.

        Returns 'self' so that other methods are available via an 'as'
        clause (though there are no interface methods at the moment).
         N/A dummy_tries N/A range N/A N/A self retries 1 N/A N/A N/A N/A N/A N/A self __acquire_locks N/A self lock_list N/A N/A N/A N/A LockError N/A N/A N/A N/A time sleep N/A self wait N/A N/A N/A LockError N/A N/A Unable to lock objects after %s tries:  N/A self retries N/A self lock_list N/A self N/A __exit__ N/A N/A self exc_type exc_value exc_traceback N/A N/A N/A Auto-called on 'with' statement exit.

        Releases the locks whether or not there has been an
        exception. Implicitly returns None which causes any
        exception to be re-raised.
         N/A object_to_unlock N/A self lock_list N/A N/A N/A N/A N/A self datacube unlock_object object_to_unlock N/A __acquire_locks N/A N/A self lock_list N/A N/A N/A Acquire all the locks on the lock_list.

        Either sucessfully acquires *all* the locks or raises a
        LockError and releases all the locks obtained so far.
         N/A lock_list N/A N/A N/A N/A N/A self datacube lock_object N/A lock_list N/A 0 N/A N/A N/A N/A N/A N/A self __acquire_locks N/A lock_list N/A 1 N/A N/A N/A N/A N/A N/A N/A self datacube unlock_object N/A lock_list N/A 0 N/A N/A N/A N/A LockError N/A N/A LockError N/A DatasetError N/A N/A Exception class used by the Lock context manager. N/A N/A