Module Expr Str ImportFrom alias ImportFrom alias alias alias alias alias alias ImportFrom alias ImportFrom alias Import alias FunctionDef arguments args defaults body Expr Str Return Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body If BoolOpAnd AttributeLoad NameLoad attr CompareEq Call AttributeLoad AttributeLoad NameLoad attr attr Str NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad Assign NameStore ListComp AttributeLoad NameLoad attr comprehension NameStore Call NameLoad NameLoad If UnaryOpNot NameLoad body Return ListLoad Return Call AttributeLoad Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad attr Call AttributeLoad NameLoad attr CompareEq AttributeLoad NameLoad attr NameLoad CompareLtE NameLoad AttributeLoad NameLoad attr CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Return Call AttributeLoad AttributeLoad NameLoad attr attr Call NameLoad keyword AttributeLoad NameLoad attr keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad Expr Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr CompareEq AttributeLoad NameLoad attr NameLoad CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr attr Call NameLoad keyword BinOpAdd NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore ListComp Call NameLoad NameLoad comprehension NameStore NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr CompareEq AttributeLoad NameLoad attr AttributeLoad NameLoad attr CompareEq AttributeLoad NameLoad attr NameLoad attr Assign NameStore Call NameLoad For NameStore NameLoad body If BoolOpAnd AttributeLoad NameLoad attr CompareIn Str AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Return Call NameLoad ListComp TupleLoad NameLoad Call NameLoad NameLoad comprehension NameStore NameLoad decorator_list FunctionDef arguments args defaults body Expr Str Assign NameStore Call NameLoad Expr Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr CompareLt AttributeLoad NameLoad attr NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list
N/A N/A Utilities related to cached snapshots and garbage collection logic.

The snapshot garbage collection service works by maintaining a master
list of which snapshots a given cluster should use. There are only two
operations that can be made to the the garbage collection api: sync
and update.

Syncing to the garbage collection is done by sending slaves on a given
cluster the list of snapshot ids they are responsible for having. The
slave is then required to download those snapshots and is free to
remove any not in that list. This corresponds to get_cached_snapshot_images.

Each cached snapshot image at some point will expire after it has been
superceded by a new image (ie, some snapshot was created for the same
project since the original snapshot). When this occurs, the snapshot
image is assigned an expiration date based on the configuration value
SNAPSHOT_EXPIRATION_DELAY (in hours). Lists of snapshot images sent
to slaves will never contain expired snapshot images.

Updating the garbage collection is append-only. When a new snapshot
is added to the cached snapshot system, it automatically sets the
expiration date for the old snapshot images under the same project
and gives the new snapshot no expiration date (it will only receive
one once it has been superceded).

The SNAPSHOT_EXPIRATION_DELAY is used because a constant time has no
real effect as long as it isn't too large and because there may be
builds running still that are using the old images, which would make
deletion potentially unsafe without complex locking on the client side.

This api abstracts away cached snapshot images so that no other parts of
changes have to know about their existance as a table separate from
snapshot imagese.
 changes.config db changes.models CachedSnapshotImage Plan PlanStatus Snapshot SnapshotImage Step datetime datetime flask current_app N/A sqlalchemy get_current_datetime N/A N/A N/A N/A N/A Gets the current datettime. Equivalent to datettime.utcnow().

    We can't mock datetime - so we abstract to this.
     N/A N/A N/A datetime utcnow N/A get_plans_for_cluster N/A N/A cluster N/A N/A N/A Returns the set of active plans for the given cluster.
     N/A plans N/A set N/A q N/A N/A N/A db session query Plan Step N/A q N/A N/A q filter N/A N/A Plan id N/A Step plan_id N/A N/A Plan status N/A PlanStatus active N/A N/A plan step N/A N/A q all N/A N/A N/A N/A step data N/A N/A N/A N/A step data get cluster None cluster N/A N/A N/A N/A plans add plan N/A plans N/A get_cached_snapshot_images N/A N/A cluster N/A N/A N/A Get the cached snapshot images for a specific cluster.

    Return all of the non-expired snapshot images for a specific
    cluster by checking all build plans to see which plans the cluster
    is associated with and grabbing snapshot images for any unexpired
    image associated with those plans.
     N/A now N/A get_current_datetime N/A plan_ids N/A N/A plan id N/A plan N/A get_plans_for_cluster cluster N/A N/A plan_ids N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A db session query SnapshotImage filter N/A N/A sqlalchemy or_ N/A N/A CachedSnapshotImage expiration_date None N/A now N/A CachedSnapshotImage expiration_date N/A N/A CachedSnapshotImage id N/A SnapshotImage id N/A N/A N/A SnapshotImage plan_id in_ plan_ids all N/A _cache_image N/A N/A snapshot_image N/A N/A N/A Creates a cached image for a given snapshot image.

    We transparently update any existing images with the same id
    in case we are caching a snaphsot that used to be cached but is
    no longer cached. This is useful for example if a snapshot is
    determined to not work and so you need to rewind to a previous
    snapshot that has already expired.
     N/A N/A N/A N/A db session merge N/A CachedSnapshotImage id N/A snapshot_image id expiration_date None N/A cache_snapshot N/A N/A snapshot N/A N/A N/A Update the cache with a newly generated snapshot.

    Caches a new snapshot (by caching all of its images).
    This creates expirations for all cached
    snapshots with no expiration and adds a new cached snapshot
    with no expiration.
     N/A now N/A get_current_datetime N/A N/A N/A N/A N/A N/A CachedSnapshotImage query filter N/A N/A CachedSnapshotImage expiration_date None N/A N/A CachedSnapshotImage id N/A SnapshotImage id N/A N/A SnapshotImage snapshot_id N/A Snapshot id N/A N/A Snapshot project_id N/A snapshot project_id update N/A dict expiration_date N/A now N/A N/A current_app config N/A CACHED_SNAPSHOT_EXPIRATION_DELTA N/A images N/A N/A N/A SnapshotImage query filter N/A N/A SnapshotImage snapshot_id N/A snapshot id N/A cached_snapshot_images N/A N/A _cache_image image N/A image images N/A N/A N/A N/A db session add_all cached_snapshot_images N/A N/A N/A N/A db session commit N/A get_relevant_snapshot_images N/A N/A snapshot_id N/A N/A N/A Given a snapshot id, detect the relevant clusters and find their
    cached snapshot images.

    Returns a dict whose keys are clusters that need to be synced
    based on the new snapshot and values are the list of snapshot
    images associated with the snapshot.
     N/A steps N/A N/A N/A N/A N/A Step query filter N/A N/A Plan id N/A SnapshotImage plan_id N/A N/A Step plan_id N/A Plan id N/A N/A SnapshotImage snapshot_id snapshot_id all N/A clusters N/A set N/A step steps N/A N/A N/A N/A step data N/A cluster N/A step data N/A N/A N/A N/A clusters add N/A N/A step data N/A cluster N/A N/A dict N/A N/A cluster N/A get_cached_snapshot_images cluster N/A cluster clusters N/A clear_expired N/A N/A N/A N/A N/A Utility function that deletes all expired rows.
     N/A now N/A get_current_datetime N/A N/A N/A N/A N/A N/A CachedSnapshotImage query filter N/A N/A CachedSnapshotImage expiration_date now delete N/A N/A N/A N/A db session commit N/A