Repository: kubernetes
Issue: API clients should be able to distinguish "invalid transition" validation errors from "invalid content" errors
Labels: sig/api-machinery,kind/feature,sig/architecture,lifecycle/frozen,triage/accepted

### What would you like to be added?

When validating update requests, the API server should indicate in returned errors when they are due to a forbidden  transition from the old object, rather than inherently invalid content in the new object.

User story 1: If I submit an update request trying to change an immutable field to an otherwise valid value, the returned error should indicate the error was specific to an update of an existing object in a programmatically discernible way.

User story 2: If I submit an update request trying to clear an allocated field that is allowed to be unset in create requests, the returned error should indicate the error was specific to an update of an existing object in a programmatically discernible way.

The API server should not skip validation of new objects in update requests just because it is also checking immutability.

User story 3: If I submit an update request trying to change an immutable field to an invalid value, I should receive an error indicating the value is not valid, and also possibly the error from user story 1 indicating the change is not allowed. I should not *only* get the error from user story 1.

---

Background:

The API server validates update requests by checking two things:
* the new object passes validation rules
* the new object does not make any forbidden transitions for field values compared to the existing object

Examples of forbidden transitions:

* Trying to change immutable fields like pod `.spec` (https://grep.app/search?q=ValidateImmutableField&filter[repo][0]=kubernetes/kubernetes)
* Trying to change or clear allocated fields like service `.spec.clusterIP`

When an update is rejected because of these errors, the returned errors are currently indistinguishable from errors indicating the updated object is invalid independent of the existing object.

### Why is this needed?

Clients that perform declarative reconciliation need to be able to distinguish between "inherently invalid" errors and "invalid relative to existing object" errors.

`kubectl apply` allows callers to specify `--force` to use the new object even if that means deleting the existing object.

However, kubectl can't tell the difference between an invalid error caused because the new object is actually invalid, and an error caused because the new object tries to perform an invalid transition.

Because it can't tell, when invoked with `--force`, `kubectl apply` responds to *any* Invalid or Conflict error with a speculative delete of the existing object, followed by an attempt to create the new object:

https://github.com/kubernetes/kubernetes/blob/cf9b5ab95a21e5b32b6aff72492a18ede37b785b/staging/src/k8s.io/kubectl/pkg/cmd/apply/patcher.go#L201-L210

If the new create fails, it then tries to recover by recreating the existing object. However, even if that recreate succeeds, ripple effects from the deletion might not be recoverable (e.g. garbage collection of dependents, etc):

https://github.com/kubernetes/kubernetes/blob/cf9b5ab95a21e5b32b6aff72492a18ede37b785b/staging/src/k8s.io/kubectl/pkg/cmd/apply/patcher.go#L224-L228

This is a generalization of the discussion around [handling of immutable fields in a declarative API].