Repository: kubernetes
Issue: RawExtension yaml/json doesn't roundtrip to equal objects - causes ControllerRevisions that can break update validation
Labels: kind/bug,priority/important-soon,sig/api-machinery,lifecycle/frozen

/sig api-machinery
/kind bug

@kubernetes/sig-api-machinery-bugs 

One controller revision can be un-updatable if its content's order isn't forced by k/k codec stack

**What happened**:

Initially I found some error logs in garbage-collector/kube-controller-manager which was trying to orphaning a ControllerRevision to perform a cascading deletion:


> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"ControllerRevision.apps \"holiday-78b7ffcd54\" is invalid: data: Invalid value: runtime.Unknown{TypeMeta:runtime.TypeMeta{APIVersion:\"\", Kind:\"\"}, Raw:[]uint8{0x7b, 0x22, 0x61, 0x66, 0x66, 0x69, 0x6e, 0x69, 0x74, 0x79, 0x22.....<omitted>... ContentEncoding:\"\", ContentType:\"application/json\"}: field is immutable","field":"data"}]},"code":422}

but actually it wasn't trying to update any content but `ownerReferences` in that controller revision, and the patch request body was:

> {"metadata":{"ownerReferences":[{"$patch":"delete","uid":"e620de7e-26aa-11e9-9276-00163e000b8d"}],"uid":"e628e563-26aa-11e9-9e54-00163e000652"}} 

After digging into the error and i find that the update validation is failing b/c the data content in controller revision is re-ordered when it's deserialized from storage. here i caught their dumps by adding some logs:

the old controllerRevision.Data in the failing update validation:

> "{"containers":[{"name":"holiday","image":":20190130003346413-master.01fc4fb8","resources"
:{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"500m","memory":"512Mi"}},"terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","
terminationGracePeriodSeconds":30,"dnsPolicy":"Default","securityContext":{},"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpre
ssions":[{"key":"failure-domain.beta.kubernetes.io/cell","operator":"In","values":["",""]}]}]}}},"schedulerName":"default-scheduler"}"

the new controllerRevision.Data in the failing update validation:

> {"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"failure
-domain.beta.kubernetes.io/cell","operator":"In","values":["",""]}]}]}}},"containers":[{"image":":2019013000
3346413-master.01fc4fb8","imagePullPolicy":"Always","name":"holiday","resources":{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"500m","memory":"512Mi"}},"terminati
onMessagePolicy":"File"}],"dnsPolicy":"Default","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30}"

The content of that controller revision wasn't decoded by k/k's codec stack, and it's just roughly decoded by calling `json.Unmarshal`. so the `containers` field is placed before the `affinity` but when it is deserialized again by apiserver in memory, the order changed as is shown above.

**How to reproduce it (as minimally and precisely as possible)**:

Just create a controller revision w/ direct HTTP call (instead of creating via `kubectl`) and w/ a json body:

```json
{
  "apiVersion": "apps/v1",
  "kind": "ControllerRevision",
  "metadata": {
    "name": "test2-holiday-78b7ffcd54",
    "namespace": "default"
  },
  "data": {
    "containers": "foo",
    "affinity": "bar"
  },
  "revision": 1
}
```

👿then any update on this object will fail due to the order forced by k/k codec stack