Extend your Kubernetes APIs with CRDs
Tags: devops, tutorial, productivity, litmuschaos
Published: 2021-03-01 13:09:12

After providing developers with a platform to automate deployment, scaling and operations of application containers across the host cluster, **Kubernetes** has really cemented its way to become a *de-facto standard* when it comes to *container-orchestration*.

With Kubernetes, you get the flexibility to roll out updates more efficiently, at whichever time and also divert the traffic to the new version if needed. It provides you options to scale your application up and down based on your requirement, you get to choose how your application interacts with other applications or the real world.

In this article, we'll take a look at:
* What is a CRD
* How do they work
* How you can create one!
* How to delete CRDs
* Examples of real-life CRDs in use within **Litmus**

By the end of this article, you'd have a thorough understanding of what are CRs, how CRDs work and how **Litmus** is also implementing them for their daily use-case.

---

## Resource and Custom Resource

Kubernetes makes it possible to provide a plethora of features like this by leveraging what is known as a **resource**. A **resource** is an endpoint in k8s API that allows you to store an API object of any kind. But what if developers need a custom object or resource based on their specific requirements? This is where a **Custom Resource** comes into the picture. 

As a part of the Kubernetes 1.7 release, they introduced the concept of Custom Resources to extend the capabilities by adding any kind of API object useful for your application. **Custom Resource Definition(CRD)** is what you use to define a Custom Resource. This is a powerful way to extend Kubernetes capabilities beyond the default installation.

---

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bsgo5rcqgfnj3whiqafi.png) 

## How does a Custom Resource Work?

As mentioned above, pretty much everything we create in Kubernetes is a resource. A Pod, a Service, a Deployment or a Secret, it's all at its very basic form a resource. These resources are often monitored by **Controllers** that are responsible for taking the information in the resource and turning it into something useful.

Let's take the example of what happens when you create a Pod resource. Whenever you modify the Pod resource i.e you add, change or update the resource, there is a **controller** that gets notified about the change. Whenever the controller sees a Pod resource being added or updated, it looks up the information in the resource and then takes a decision on what to do next. In the case of a new Pod, it'll figure out what node is responsible for running the workload and assign it to that node. This assignment is then picked up by another controller on that node, which in turn makes sure to spin up the required containers.

Custom resources are defined using the custom resource definition and that's why the abbreviation CRD and not CR. Once CRD is created, the Kubernetes API server creates a RESTful path to each version specified in the CRD. Based on the scope defined on the CRD file, the path may either be accessible globally by the entire cluster or only to a specific project. A CR, once defined by CRD, is then stored in **etcd** cluster. Kubernetes API server takes care of the resource lifecycle and replication. It acts similar to the native resources and when a project is deleted, all the custom resources and native resources get deleted as well.

---

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pi8g9erib0q895ek1lr1.png) 

## Creating a CRD

As a pre-requisite, you'd need Kubernetes 1.7 or above and a kubectl tool installed to be able to create CRDs. Just like a lot of other files in the Kubernetes world CRDs can also be created with the help of YAML files. A CRD YAML file consists of several fields, some of the important once are listed down:

* **metadata.name**: Name of the CRD that you're creating
* **spec.group**: Name of the group the CRD object belongs to.
* **spec.versions**: It holds the different versions of custom resources that will be created.
* **spec.names**: Used to define how to describe your Custom Resource. You can add plural, singular and short names of your CR here which can later be handy while managing it through kubectl.
* **spec.names.kind**: Defines the kind of CR that can be created using the CRD. Like **Deployment**,**CronJob**,**CronTab**.
* **spec.scope**: Defines what is the scope of the CR. It can either be set to 'clustered' or 'namespaced'. It is set to 'namespaced' by default.

The manifest below shows an example CRD `ourcrd.yaml`:

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crdname.demo.example.com
spec:
  scope: Namespaced
  group: demos.example.com
  names:
    kind: Foo
    singular: crdname
    plural: crdnames
    shortNames:
    - cn
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          description: "A foo object"
          properties:
            spec:
              type: object
              properties:
                value1:
                  type: string
                  default: "Default value"
                value2:
                  type: integer
```

We can create the above CRD simply using
```bash
kubectl create -f ourcrd.yaml
```

After creating the CRD through kubectl, the endpoint URL is generated that can be used to create and manage custom objects. This endpoint might take a couple of seconds to get created.

You might have this question, "Okay! That cool, but how/where would I possibly use this?", well, here is an example of how you can use it.

Let's create a manifest using the kind we created with the CRD. For example, we have `my-kind.yaml`

```yaml
apiVersion: "demo.example.com/v1"
kind: Foo [The same kind we declared in the CRD]
metadata:
  name: demo-app
spec:
  value1: "Hello Developers"
  value2: 92
```

Here we are using the same kind we defined in our CRD, we then define the fields we want our kind object to have. Like in this example, we gave the fields as value1 and value2(As provided in the CRD) for our app. In real examples, these fields would be something like *image*, *URI*, etc.

Now if we run `kubectl create -f my-kind.yaml` our application can consume the data we created with the manifest above. To see what is going on we can run `kubectl get cn -o yaml`, we can see detailed info on what we just created. 

> cn is the short name we gave to our CRD above

---

## Deleting a CRD

To delete the CRD and resources we created, simply run kubectl delete just like with any other resources. When you delete the CRD, the server uninstalls the RESTful endpoint that was created with the CRD. With the deletion of a CRD, all the custom resources created using it are also deleted and cannot be retrieved.

---

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cyq6npd5ktnv0nzp7so4.png) 

## Real Life example of CRDs

In this last section of our journey let's actually take a look at how real projects utilize these concepts to expand upon their requirements and actually implement them.

I'm going to take the example of Litmus as the project whose CRDs we are going to look into.

![Litmus](https://res.cloudinary.com/practicaldev/image/fetch/s--jU91rNXg--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/9a1qur50nxb8hlp8w0nk.png)

Litmus is a framework for practising chaos engineering in [cloud-native environments](https://dev.to/umamukkara/chaos-engineering-for-cloud-native-systems-2fjn). Litmus provides a chaos operator, a large set of chaos experiments on its [hub](https://hub.litmuschaos.io/), detailed [documentation](https://docs.litmuschaos.io/docs/getting-started/installation), and a friendly [community](https://litmuschaos.io/). Litmus is very easy to use; you can also set up a very [quick demo environment](https://dev.to/uditgaurav/get-started-with-litmuschaos-in-minutes-4ke1) to install and run Litmus experiments.

Few of the important CRDs in Litmus are as follows:

* ChaosEngine
* ChaosExperiment
* ChaosResult

**ChaosEngine**: ChaosEngine CR is created for a given application and is tagged with appLabel. This CR ties one or more ChaosExperiments to an application.

```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: chaosengines.litmuschaos.io
spec:
  group: litmuschaos.io
  names:
    kind: ChaosEngine
    listKind: ChaosEngineList
    plural: chaosengines
    singular: chaosengine
  scope: Namespaced
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          type: object
          properties:
            monitoring:
              type: boolean
            jobCleanUpPolicy:
              type: string
              pattern: ^(delete|retain)$
              # alternate ways to do this in case of complex pattern matches
              #oneOf:
              #  - pattern: '^delete$'
              #  - pattern: '^retain$'
            annotationCheck:
              type: string
              pattern: ^(true|false)$
            appinfo:
              type: object
              properties:
                appkind:
                  type: string
                  pattern: ^(^$|deployment|statefulset|daemonset|deploymentconfig|rollout)$
                applabel:
                  type: string
                appns:
                  type: string
            auxiliaryAppInfo:
              type: string
            engineState:
              type: string
              pattern: ^(active|stop)$
            chaosServiceAccount:
              type: string
            components:
              ...
            experiments:
              type: array
              items:
                 ...
        status:
          type: object
  version: v1alpha1
  versions:
    - name: v1alpha1
      served: true
      storage: true
```

**ChaosExperiment**: ChaosExperiment CR is created to hold and operate the details of actual chaos on an application. It defines the type of experiment and key parameters of the experiment.

```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: chaosexperiments.litmuschaos.io
spec:
  group: litmuschaos.io
  names:
    kind: ChaosExperiment
    listKind: ChaosExperimentList
    plural: chaosexperiments
    singular: chaosexperiment
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          type: object
          properties:
            ...
        status:
          type: object
  version: v1alpha1
  versions:
    - name: v1alpha1
      served: true
      storage: true
```

**ChaosResult**: ChaosResult CR is created by the operator after an experiment is run. One ChaosResult CR is maintained per ChaosEngine. The ChaosResult CR is useful in making sense of a given ChaosExperiment. This CR is used for generating chaos analytics which can be extremely useful — for example when certain components are upgraded between the chaos experiments, and the results need to be easily compared

```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: chaosresults.litmuschaos.io
spec:
  group: litmuschaos.io
  names:
    kind: ChaosResult
    listKind: ChaosResultList
    plural: chaosresults
    singular: chaosresult
  scope: Namespaced
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          type: object
        status:
          type: object
  version: v1alpha1
  versions:
    - name: v1alpha1
      served: true
      storage: true
```

> To take a wider look at the CRs [visit here](https://github.com/litmuschaos/litmus/blob/master/litmus-portal/litmus-portal-crds.yml). You'd find the CRDs of Cluster Workflows, Cron Workflows, etc

---

## Conclusion

Voila, you have successfully learned about the concept of CRDs and how to create them. You can of course extend these and play around to create something out of your own. We welcome everyone to comment and let us know about what/how we can improve these configurations to achieve more! Every suggestion is appreciated.

A custom resource definition adds to the incredible features Kubernetes already offers to its users. CRD helps extend Kubernetes’ features helps make it an even more universal tool for container orchestration. You can use custom resources to add your own resources that help with your specific requirements. And, you can use these resources like any native Kubernetes service and leverage all the features Kubernetes has to offer like security, RBAC, API services, and CLI. You can also use dynamic registration to have the custom resources appear and disappear while the cluster is running.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n0l4sshdm3c4w51vpvqe.png) 

Are you an [SRE](https://en.wikipedia.org/wiki/Site_Reliability_Engineering) or a [Kubernetes](https://kubernetes.io/) enthusiast? Does [Chaos Engineering](https://en.wikipedia.org/wiki/Chaos_engineering) excite you?
Join Our Community On Slack For Detailed Discussion, Feedback & Regular Updates On Chaos Engineering For Kubernetes: https://kubernetes.slack.com/messages/CNXNB0ZTN
(#litmus channel on the Kubernetes workspace)

<p align="center">
  <img src="https://i.ibb.co/XtpJ3Wz/Litmus.png" width="1500" />

Check out the [Litmus Chaos GitHub repo](https://github.com/litmuschaos/litmus) and do share your feedback: https://github.com/litmuschaos/litmus
Submit a [pull request](https://github.com/litmuschaos/litmus/pulls) if you identify any necessary changes.

Don't forget to share these resources with someone who you think might benefit from them. Peace out. ✌🏼