Understanding the controller-runtime Cache Seriously
Tags: kubernetes, kubebuilder, operator
Published: 2024-05-11 00:37:06

---
title: Understanding the controller-runtime Cache Seriously
published: true
description: Understand controller-runtime Cache deeply.
tags: kubernetes,kubebuilder,operator
# cover_image: https://direct_url_to_image.jpg
# Use a ratio of 100:42 for best results.
# published_at: 2024-05-11 00:18 +0000
---

The [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime/tree/main) package provides a [Cache](https://github.com/kubernetes-sigs/controller-runtime/tree/24cc51fde1b37ece947b6ecc4f92ceaa4fe942b5/pkg/cache) mechanism to users so that they don need to be aware of Informer's presence. By default, when you Get/List Kubernetes Objects, the Cache will automatically start Informers in the background and cache all Objects with the same GVK (GroupVersionKind).

This (somewhat naïve) behavior of the Cache usually doesn't pose an issue in small Kubernetes Clusters. However, as the cluster scales up, this can lead Controllers to unexpectedly consume a significant amount of memory. To address this, the Cache comes equipped with [Options](https://github.com/kubernetes-sigs/controller-runtime/blob/24cc51fde1b37ece947b6ecc4f92ceaa4fe942b5/pkg/cache/cache.go#L135) to finely control the behavior of Informers.

However, there is a lack of documentation regarding these Options, making it challenging to understand the intricate behaviors. Therefore, we will first delve into the Cache code and then consider the behavior of the Options. When we refer to "Cache" below, it encompasses the [Cache interface](https://github.com/kubernetes-sigs/controller-runtime/blob/24cc51fde1b37ece947b6ecc4f92ceaa4fe942b5/pkg/cache/cache.go#L65), as well as its implementations, namely [informerCache](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/informer_cache.go#L67), [multiNamespaceCache](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/multi_namespace_cache.go#L68), and [delegatingByGVKCache](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/delegating_by_gvk_cache.go#L33C6-L33C26).

## Get/List and Cache
Before getting into the details of Cache, let's check the general flow of Get/List Kubernetes Objects using Cache. When using controller-runtime, you typically start by obtaining a Client to interact with the Kubernetes API using the [GetClient method](https://github.com/kubernetes-sigs/controller-runtime/blob/a6b9c0b672c77a79fff4d5bc03221af1e1fe21fa/pkg/cluster/cluster.go#L57).

The actual GetClient method is [controllerManager's GetClient method](https://github.com/kubernetes-sigs/controller-runtime/blob/77435913f5134b524dc0b83e8ed754a3f4174281/pkg/manager/internal.go#L224), which internally calls the [cluster's GetClient method](https://github.com/kubernetes-sigs/controller-runtime/blob/4fd4f6e7116453ea4e07a2cdd8ba0d38ad64680f/pkg/cluster/internal.go#L70 This method simply returns the value of the cluster's client field, so we'll now look at what is set in this client field.

What is set in the client field is [clientWriter](https://github.com/kubernetes-sigs/controller-runtime/blob/a6b9c0b672c77a79fff4d5bc03221af1e1fe21fa/pkg/cluster/cluster.go#L227). The `clientWriter` is initialized using the options.NewClient function, with the [client package's New function](https://github.com/kubernetes-sigs/controller-runtime/blob/55c4331ea389d4d0332ac6d01b9b1c02969a62/pkg/client/client.go#L109) set as the default. Here, Cache is also initialized using the options.NewCache function, with the [cache package's New function](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L307) set as the default. The initialized Cache is passed as an argument to [options.NewClient](https://github.com/kubernetes-sigs/controller-runtime/blob/a6b9c0b672c77a79fff4d5bc03221af1e1fe21fa/pkg/cluster/cluster.go#L224)[^1][^2].

The cache package's New function then calls the [newClient function](https://github.com/kubernetes-sigs/controller-runtime/blob/55c4331eaec389d4d0332ac6d01b9b1c02969a62/pkg/client/client.go#L117) to initialize the Client. It is clear that the Client's cache field is set with the Cache we initialized earlier [(link)](https://github.com/kubernetes-sigs/controller-runtime/blob/55c4331eaec389d4d0332ac6d01b9b1c02969a62/pkg/client/client.go#L203).

All we need to do now is to look at the client's [Get](https://github.com/kubernetes-sigs/controller-runtime/blob/55c4331eaec389d4d0332ac6d01b9bc02969a62/pkg/client/client.go#L352-L357) and [List](https://github.com/kubernetes-sigs/controller-runtime/blob/55c4331eaec389d4d0332ac6d01b9b1c02969a62/pkg/client/client.go#L374-L379) methods. After calling the [shouldBypassCache method](https://github.com/kubernetes-sigs/controller-runtime/blob/55c4331eaec389d4d0332ac6d01b9b1c02969a62/pkg/client/client.go#L235), if the Kubernetes Object is allowed to be cached, the Get/List methods of Cache are called.

Although this was rather extensive, we've now determined that calling Get/List using a client obtained from the GetClient method actually calls the Get/List methods of Cache.

## Types of Cache
The following types of Cache implement the [Cache interface](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L65):

1. [informerCache](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/informer_cache.go#L67)
2. [multiNamespaceCache](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/multi_namespace_cache.go#L68C6-L68C26)
3. [delegatingByGVKCache](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/delegating_by_gvk_cache.go#L33C6-L33C26)

informerCache is the most basic implementation of Cache, and it is directly used as Cache without specifying the Cache Options DefaultNamespaces or ByObject. multiNamespaceCache is used when setting the [DefaultNamespaces option](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L192), and internally has a map of [namespace name to Cache (= informerCache)](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/multi_namespace_cache.go#L72), allowing you to specify the behavior of Informers for particular namespaces.

delegatingByGVKCache is used when setting the [ByObject option](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L227), and internally a map of [GVK to Cache](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/delegating_by_gvk_cache.go#L35). It allows you to specify the behavior of Informers for particular GVKs. If the [Namespaces option of ByObject](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L255) is set[^3], then multiNamespaceCache is used as Cache for the corresponding GVK; otherwise, informerCache is used. Let's take a closer look at the behavior of each Cache.

### informerCache

Using the [informerCache's Get method](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/informer_cache.go#L74) as an example, let's confirm the flow of dynamically created Informers. The Get method internally calls the [getInformerForKind method of informerCache](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/informer_cache.go#L80), which in [calls the Get method of Informers](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/informer_cache.go#L190). If the Informer does not exist, Get method of Informers invokes the [addInformerToMap method](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/internal/informers.go#L342) to [create an Informer](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/internal/informers.go#L358-L370).

The created Informer is packed into a Cache entry (this Cache is not the Cache interface of the [cache package](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/cache.go#L65), but rather the [Cache struct of the internal package](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/internal/informers.go#L87)), and is then [saved to the tracker field of Informers](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/internal/informers.go#L138) via the [informersByType method](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/internal/informers.go#L330)[^4].

The Get method of informerCache eventually obtains the Object through the [Get method of CacheReader set in the Reader field of this Cache entry](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/informer_cache.go#L88). The [Get method of CacheReader](https://github.com/kubernetes-sigs/controller-runtime/blob/bbcaf6a9d9ff0ba256e18800f194d387b96dca33/pkg/cache/internal/cache_reader.go#L57) retrieves the Object from the [Indexer](https://github.com/kubernetes-sigs/controller-runtime/blob/bbcaf6a9d9ff0ba256e18800f194d387b96dca33/pkg/cache/internal/cache_reader.go#L64).

### multiNamespaceCache
For multiNamespaceCache as well, let's quickly look at its behavior. The multiNamespaceCache uses the [newMultiNamespaceCache function](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/multi_namespace_cache.go#L38) to [create an informerCache for each namespace](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/multi_namespace_cache.go#L48) and saves them in the namespaceToCache field. The [Get method of multiNamespaceCache](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/multi_namespace_cache.go#L226) extracts the corresponding informerCache from the namespaceToCache field based on the namespace of the given Object, and [calls the Get method of that informerCache](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/multi_namespace_cache.go#L244).

### delegatingByGVKCache
The delegatingByGVKCache [generates a Cache for each GVK based on the ByObject option](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L333-L345) and [saves them in the caches field of delegatingByGVKCache](https://github.com/kubernetes-sigs/controller-runtime/blob/24cc51fde1b37ece947b6ecc4f92ceaa4fe942b5/pkg/cache/delegating_by_gvk_cache.go#L35). As mentioned before, the Cache created here will be either an informerCache or a multiNamespaceCache depending on the option. The [Get method of delegatingByGVKCache](https://github.com/kubernetes-sigs/controller-runtime/blob/bbcaf6a9d9ff0ba256e18800f194d387b96dca33/pkg/cache/delegating_by_gvk_cache.go#L39) extracts the Cache corresponding to the given Object's GVK and [calls the Get method of that Cache](https://github.com/kubernetes-sigs/controller-runtime/blob/bbcaf6a9d9ff0ba256e18800f194d387b96dca33/pkg/cache/delegating_by_gvk_cache.go#L44).

## Builder's For, Watches, and Owns, and Cache
In addition to calling Client's Get/List, you can also register Informers through Builder's [For](https://github.com/kubernetes-sigs/controller-runtime/blob/2add01e784f9f75a8f1be38fc0a21eee5eaee84f/pkg/builder/controller.go#L84), [Watches](https://github.com/kubernetes-sigs/controller-runtime/blob/2add01e784f9f75a8f1be38fc0a21eee5eaee84f/pkg/builder/controller.go#L137), and [Owns](https://github.com/kubernetes-sigs/controller-runtime/blob/2add01e784f9f75a8f1be38fc0a21eee5eaee84f/pkg/builder/controller.go#L114) methods. Let's check the behavior of Cache in these cases as well. If we look at the [Builder's doWatch method](https://github.com/kubernetes-sigs/controller-runtime/blob/2add01e784f9f75a8f1be38fc0a21eee5eaee84f/pkg/builder/controller.go#L275), whether you use For, Watches, or Owns, you ultimately call the [Watch method of the Controller](https://github.com/kubernetes-sigs/controller-runtime/blob/5d388e9c8380816fa655fd3b6999b8b04ba5cd4c/pkg/internal/controller/controller.go#L118), which then [uses the Start method of Kind](https://github.com/kubernetes-sigs/controller-runtime/blob/2add01e784f9f75a8f1be38fc0a21eee5eaee84f/pkg/internal/source/kind.go#L41) to [register the Informer using the GetInformer method of Cache](https://github.com/kubernetes-sigs/controller-runtime/blob/2add01e784f9f75a8f1be38fc0a21eee5eaee84f/pkg/internal/source/kind.go#L66).

## Cache Options

Let's take a closer look at the behaviors of [Cache Options](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L135).

### ReaderFailOnMissingInformer
[ReaderFailOnMissingInformer](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L179) is an option that prevents the dynamic start-up of Informers through Get/List actions. By looking at the [getInformerForKind method of informerCache](https://github.com/kubernetes-sigs/controller-runtime/blob/bc35946fac1b56071def3ebf299996b281c77442/pkg/cache/informer_cache.go#L181), we can see that it will return an error if there is no corresponding Informer for an Object's GVK. You can preemptively start Informers either by calling Cache's GetInformer method through Builder's For, Watches, and Owns methods, or by directly invoking Cache's GetInformer method.

### DefaultNamespaces
[DefaultNamespaces](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L192) is an option for using multiNamespaceCache to specify Informer behavior per namespace. To see what options can be specified with DefaultNamespaces, check out the [Config](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L278). Let's consider a few edge cases when DefaultNamespaces is specified.

#### Get/List for Cluster-scoped Object
Upon reviewing the [Get method of multiNamespaceCache](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/multi_namespace_cache.go#L226), it is clear that if an Object is cluster-scoped, then it calls the Get of `clusterCache`. The `clusterCache` is an [informerCache without namespace restriction](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/multi_namespace_cache.go#L54)[^5]. This means that even if DefaultNamespaces is specified, Informers for cluster-scoped Objects can be started using `clusterCache`, and Get/List operations can be performed.

#### Get/List for Objects Outside the Specified Namespace
When we revisit the [Get method of multiNamespaceCache](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/multi_namespace_cache.go#L226), the first step is to search `namespaceToCache` using the namespace of the provided Object. If a Cache is not found, it searches `namespaceToCache` again using `metav1.NamespaceAll` (=""). This indicates that for Get/List of Objects outside the specified Namespace, if the empty string namespace is specified in DefaultNamespaces, informerCache is utilized; otherwise, an error is returned.

### DefaultTransform
[DefaultTransform](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L207) specifies the default Transform function to pass to Informers. The Transform function allows modifications to be made to an Object before it is processed by the Informer. A common use case includes deleting fields from an Object (such as Metadata) to reduce memory consumption[^6].

### DefaultUnsafeDisableDeepCopy
[DefaultUnsafeDisableDeepCopy](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L223) is an option to conserve memory usage when calling Get/List by allowing the CacheReader to return Kubernetes Objects cached by the Informer without performing a DeepCopy. As stated in the [comment](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L218-L219), enabling this option requires you to perform a DeepCopy when making changes to the Object. This option is unlikely to be enabled unless there are special circumstances, such as listing a large number of Objects in a single reconcile.

### ByObject
[ByObject](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L227) is an option to use delegatingByGVKCache to specify the behavior of Informers for each GVK. The available options that can be specified with ByObject can be found in the [ByObject struct](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L234). This option is similar to DefaultNamespaces, but the behavior is different when an Object with a GVK not specified by ByObject is passed to Get/List.

Unlike DefaultNamespaces, which returns an error for most unspecified Namespace Get/List operations, the [delegatingByGVKCache's cacheForGVK method](https://github.com/kubernetes-sigs/controller-runtime/blob/f4dbd14c2db27a4f5a700c1c0e96dc0a0b18447f/pkg/cache/delegating_by_gvk_cache.go#L129) returns `dbt.defaultCache` when the provided GVK is not in `dbt.caches`.

The `defaultCache` will be either a multiNamespaceCache or an informerCache depending on [other options](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L316-L321). Thus, with ByObject, if an Object with an unspecified GVK is provided, an error typically does not return, and an Informer may be dynamically started.

In addition to the above, there are other Cache Options such as [SyncPeriod](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L170C2-L170C12), [DefaultLabelSelector](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L196), [DefaultFieldSelector](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L200C2-L200C22), as well as the [Config from DefaultNamespaces](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L278) and the [ByObject option](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L234). However, we won't delve into these here as their behaviors can be easily understood from the discussion so far.

[^1]: The initialized Cache is [set to the cluster's cache field](https://github.com/kubernetes-sigs/controller-runtime/blob/a6b9c0b672c77a79fff4d5bc03221af1e1fe21fa/pkg/cluster/cluster.go#L254), from where it is [started through the cluster's Start method](https://github.com/kubernetes-sigs/controller-runtime/blob/4fd4f6e7116453ea4e07a2cdd8ba0d38ad64680f/pkg/cluster/internal.go#L104). The cluster itself is [added as a Runnable to the controllerManager's runnables](https://github.com/kubernetes-sigs/controller-runtime/blob/58719057609722d090ac4ee3f1866e5958d16e6f/pkg/manager/internal.go#L356) and started. The concrete implementation of the Start method held by the Cache interface is the [Informers' Start method](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/internal/informers.go#L189) for the informerCache, whereas [multiNamespaceCache](https://github.com/kubernetes-sigs/controller-runtime/blob/6dd03ed1f07cf03b34d088b9056082198039e328/pkg/cache/multi_namespace_cache.go#L165) and [delegatingByGVKCache](https://github.com/kubernetes-sigs/controller-runtime/blob/f4dbd14c2db27a4f5a700c1c0e96dc0a0b18447f/pkg/cache/delegating_by_gvk_cache.go#L75) each have their own distinct Start methods.

[^2]: The [Cache](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L65) itself also implements the Start method through the [Informers interface](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L76), so it can be [directly added to the controllerManager as a Runnable](https://github.com/kubernetes-sigs/controller-runtime/blob/58719057609722d090ac4ee3f1866e5958d16e6f/pkg/manager/internal.go#L172). By creating a Client using this Cache, you can utilize a Client with Cache Options that are completely different from those of the controllerManager.

[^3]: The multiNamespaceCache is also employed when the [DefaultNamespaces option is specified](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L460). As noted in the [comment](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L247), to avoid using the DefaultNamespaces option, one must specify an empty list in the Namespaces option of ByObject.

[^4]: For more details about client-go's Informer and Indexer, refer to other documents like [client-go under the hood](https://github.com/kubernetes/sample-controller/blob/master/docs/controller-client-go.md).

[^5]: The globalConfig, when the multiNamespaceCache is generated from DefaultNamespaces, is produced using the [optionDefaultsToConfig function](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L364), but when the multiNamespaceCache is created through the delegatingByGVKCache originating from ByObject, it results in [nil](https://github.com/kubernetes-sigs/controller-runtime/blob/4cf9db06e3b06be9c57ced5d8b705339b816765d/pkg/cache/cache.go#L340).

[^6]: Another method to limit memory usage is by using [PartialObjectMetadata](https://github.com/kubernetes/apimachinery/blob/d5c9711b77ee5a0dde0fef41c9ca86a67f5ddb4e/pkg/apis/meta/v1/types.go#L1496). By utilizing PartialObjectMetadata, only the metadata of an Object is cached, which can help to reduce the amount of memory used.