How controller-runtime Cache Works and Why It Matters for Performance

by

How controller-runtime Cache Works and Why It Matters for Performance

When writing Kubernetes controllers in Go using controller-runtime, developers often operate under a fuzzy mental model: they assume r.Get() queries the API server directly and that r.List() returns a live view of the world. In practice, the model is the opposite. The cache is the foundation of the system, not just an optimization.

This guide clarifies how the controller-runtime cache operates, the architectural decisions baked into Kubernetes, and how to avoid expensive memory and performance surprises in production.

The Core Model: Reads from Memory, Writes to the API Server

The most important fact to internalize is that r.Get() and r.List() inside a reconciler typically do not read from the API server. They read from a local in-memory cache. The manager warms up this cache with an initial list and keeps it current through a watch stream.

Consequences of this design include:

  • Reads are cheap: Even hundreds of calls per second do not load the control plane.
  • Writes are exact: Create, Update, Patch, and Delete go straight to the API server.
  • Eventual consistency: After a write, there is a window where a subsequent read from the cache may still return the old state.

Internal Architecture: Reflector, Queue, and Indexer

Under the hood, the cache package is a thin wrapper around client-go primitives. The pipeline works as follows:

  1. Reflector: The only component that talks to the API server. It fetches an initial snapshot and maintains an open watch using resourceVersion.
  2. Delta Queue: Historically DeltaFIFO, but now the default is RealFIFO (since client-go 1.36). RealFIFO is a flat, ordered slice of deltas. It preserves global order and passes every event to the next stage without per-key deduplication.
  3. Indexer (Store): The in-memory object store. It holds the objects and any registered indexes.

The workqueue, which feeds your reconciler, handles deduplication at the key level, collapsing a flood of updates into a single reconcile.

Common Pitfalls and Correct Patterns

1. Read-After-Write is Not Instant

Because the cache updates asynchronously via the watch, you cannot expect to Update an object and immediately Get the new state. This is not a bug; it is an eventual consistency property. Reconcile functions must be idempotent and always re-read the current state to determine the correct action.

2. Deep Copy Behavior

The cache-backed client automatically deep-copies objects returned by Get and List. However, objects passed to Predicate or EventHandler are shared. Mutating these shared objects will corrupt the cache for other controllers. If you must mutate them, call DeepCopy() first.

3. Resync vs. Relist

A resync does not perform a list. It re-emits all current objects through the queue. A relist (fetching a fresh snapshot) only happens when the watch fails with a 410 Gone error or is explicitly recreated.

Optimizing Memory and Performance

Using Indexes (SQL-like Queries)

Without indexes, a List performs an O(n) scan over the entire store, blocking writers during the scan. You can register indexes using IndexField. For example, indexing by spec.nodeName allows you to use MatchingFields to find objects in O(log n) time. Note that MatchingLabels does not use a separate index; it still scans the store.

Selective Caching

By default, the cache pulls every object of a type from every namespace. For large clusters, this can consume gigabytes of memory. You can constrain the cache scope using cache.Options to:

  • Limit namespaces.
  • Use label selectors on the watch.
  • Apply Transform functions to drop heavy fields (like ManagedFields).

Metadata-Only Objects

If you only need object metadata (labels, names, owner references) and not the spec or data, use PartialObjectMetadata. This dramatically reduces memory usage for types like Secrets and ConfigMaps.

When to Bypass the Cache

The cache is not always the right tool. Use mgr.GetAPIReader() when:

  • You need to read objects before mgr.Start() (e.g., during initialization).
  • You need pagination using client.Continue.
  • You need a one-off read for a type you are not caching.

Additionally, you can disable caching entirely for specific types using client.Options.Cache.DisableFor, though this means you will not receive events for those objects.

Conclusion

The controller-runtime cache is the operating model, not an optimization. Understanding that reads are local and writes are remote, combined with proper use of indexes and selective caching, is essential for building performant and stable Kubernetes controllers.