Repository: kubernetes
Issue: RV vs object-wide logical clock
Labels: sig/api-machinery,triage/accepted

(followup from SIG API Machinery meeting of 2022-09-21)

(note, I'm keeping this text up to date so you don't have to read all the discussion below if you don't want to)

We are trying to unblock a class of caching behaviors, example PR: https://github.com/kubernetes/kubernetes/pull/112202

The problem is that today there is no way for a client to know which of two object states is "more recent". This makes it very hard to e.g. put the result of a write into a cache ahead of seeing the corresponding watch event.

(Aside: doing this destroys the property that listing out of cache always gives you a group of objects that actually co-existed in the server at some point in time. Not everyone needs this property, but we will probably ask that an in-tree assumption / write-through cache gives the option to restore this property. I'll refer to this as the "cross-object ordering problem".)

## Option 1: Relax prohibition on clients parsing resourceVersion.

Today we tell clients that they may compare RV as a string, only for equality. In practice, some clients actually do parse it anyway, and we already suffer the downside of not being able to change what we store in the string.

So this option is to tell clients that they may parse RV as a positive integer of up to 63 bits. The resulting integers may be compared to determine recency, but *only* between objects within the same collection (resource). RV comparison between collections is still invalid and at least some clusters are actually configured in a way that would break clients attempting this comparison.

Pros:
* People already do this. We lose almost no flexibility to change this in the future (we already lost it)
* Easy to see how this could solve the cross-object ordering problem (you can directly compare RV from objects in the same collection)
* No control plane changes needed, client changes compatible with old clusters.
 
Cons:
* Ugly; we would never do it this way if we started fresh
* 63 bits is slightly constraining for alternative DB implementations (but it's already that way)

## Option 2:
Alternatively, we could add a field to metadata, `.objectVersion`; this field would be a 63 bit int, a per-object logical clock. The server would enforce that it functions like the current `.generation` field, but for changes in the whole object, not just spec.

Pros:
* This design is much more aesthetic
* 63 bits is enough for a single object (especially since no-op changes won't increment this)
* [EDIT: perhaps in a few years we could start being more stringent about RV handling?]

Cons:
* Requires server changes, clients have to wait for a new version. ~Code has to take care for no-op updates.~ [EDIT: storage-layer changes aren't actually needed, only an api change]
* ~slight storage size increase.~ [EDIT: no, actually etcd already stores this for us]
* More difficult to solve the cross-object ordering problem (you have to wait for object X to hit at least objectVersion Y, instead of waiting to see any object with RV > Z)
* RV situation is probably not improved [EDIT: not in the short term at least].
* [EDIT: Awkward / not clear how to make this useful / play nice with other preconditions (including RV)]

## Recommendation

I recommend we select option 1, update documentation, and call it a day. [EDIT: but option 2 is back in the running after these updates...]

/sig api-machinery