Repository: kubernetes
Issue: Allow use of $ref in in CRD validation schema
Labels: sig/api-machinery,kind/feature,priority/important-longterm,area/custom-resources,lifecycle/frozen

(combined with https://github.com/kubernetes/kubernetes/issues/76965)

### `$ref` scenario 1:

In go types, we can define a type and use it in several other types. When generating CRD from this go types, we want to use reference for that type in the validation so that the schema has better readability. 
```
package v1beta1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
type DataType struct {
  Value int `json:"value,omitempty"`
}
type MyKindSpec struct {
  Data []DataType `json:"data,omitempty"`
}

type MyKindStatus struct {
  Data []DataType `json:"data,omitempty"`
}

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// MyKind
// +k8s:openapi-gen=true
// +resource:path=mykinds
type MyKind struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MyKindSpec   `json:"spec,omitempty"`
    Status MyKindStatus `json:"status,omitempty"`
}
```
**What you expected to happen**:
In the CRD for this type, we expect to see the validation as
```  
     spec:
          properties:
            data:
              items:
                $ref: '#/definitions/<...>.DataType'
```

### `$ref` scenario 2:

Original type information is lost when CRD authors [expand their openapi definitions](https://github.com/ant31/crd-validation/blob/eab5f7e90b2c/pkg/convert_types.go#L81-L89). `openapi-gen`[-based operator code is similarly affected](https://github.com/coreos/prometheus-operator/blob/master/example/prometheus-operator-crd/alertmanager.crd.yaml). CRDs forbid `$ref` (implicit GVK link) and `x-*` properties (GVK hints).

_This information enables early, developer-friendly policy checks and useful manifest transforms_. Our CICD automation extends the base k8s schema with custom policy and relies on the schema's `$ref`-erential knowledge to inspect and operate on target resource types, anywhere they appear, eg. deeply nested **or embedded within custom resources**. Prior to validation, a simple callback might first:

**ObjectMeta**
- Require teams set `app.kubernetes.io/*` labels.
- Require teams set org-specific cost/usage/alert aggregation labels.
- Back and forward compat handling, eg. `app`, `app.kubernetes.io/name`, `team`, ...

**Container**
- Inject dynamic ENV and volumes.
- Resolve string properties with templates.
- Require several `resources.limits.*` are set.

**PodSpec** ... **PodSecurityContext** ... **Volume** ...
**Deployment** ... **CronJobSpec** ... **JobSpec** ...
**ServiceSpec** ... **SecurityContext** ... _**(many more)**_ ...

`$ref` expansion makes CRDs opaque to our automation. A pre-processing step is required to repair the type information and restore visibility into CRDs. Options considered are:

- Import all golang-based `openapi-gen`'ed CRDs _directly_ (wherever they write `openapi_generated.go`) and call each `GetOpenAPIDefinitions` function. Merge all results.
- Run `openapi-gen` on k8s and CRD sources ourselves. Avoids multiple `GetOpenAPIDefinitions` calls and potential linking problems.
- Use `kube-apiserver`. Run `hack/update-openapi-spec.sh` with extra CRDs "builtin" and enabled so `GET /openapi/v2` simply returns what we need.

I'd love `$ref` support, but I understand why that could be difficult or undesirable. Alas, if CRDs permitted a known property, eg. `x-kubernetes-group-version-kind` (or similar), CRD libraries can set it accordingly for _each unique `$ref` expansion_ (no need to nest GVK hints), allowing consumers (or even k8s itself) to restore this relationship if and when useful.

Are there options we can move on?