Building a Custom Metrics Exporter for Kubernetes

by

Building a Custom Metrics Exporter for Kubernetes

Kubernetes provides built-in metrics for CPU and memory, but real-world scaling decisions often depend on signals that exist outside that narrow window: queue depth, job latency, or active WebSocket connections. When standard metrics are insufficient, a metrics exporter bridges the gap. This guide walks through building an exporter from scratch, containerizing it, and integrating it with Prometheus and the HorizontalPodAutoscaler (HPA).

What is a Metrics Exporter?

A metrics exporter is a lightweight HTTP server with a single responsibility: exposing application state as plain text on a /metrics endpoint. Prometheus scrapes this endpoint on a regular interval, stores the resulting time-series data, and makes it available for queries, alerts, and autoscaling rules.

There are two primary ways to instrument your application:

  • Embedded: Use the Prometheus client library directly within your application code to expose /metrics from the same process.
  • Standalone: Run a separate exporter process. This is preferred when the data source is external to your application or when you do not control the application code.

Choosing the Right Metric Types

Before writing code, you must decide which Prometheus metric type fits your signal. The data model has three main types:

  • Counters: Values that only ever increase (e.g., total requests served, errors encountered). Do not use counters for values that can go down.
  • Gauges: Current snapshots of values that can rise and fall freely (e.g., queue depth, active connections).
  • Histograms: Distributions of observed values, such as request latency, allowing you to calculate percentiles like p99.

Follow the naming convention <namespace>_<name>_<unit> in snake_case. For example, a job processor might expose worker_jobs_processed_total (counter), worker_queue_depth (gauge), and worker_job_duration_seconds (histogram).

Implementation with Go

The Go Prometheus client is the standard choice in the Kubernetes ecosystem. Below is a practical implementation using the client library.

Project Setup

Initialize a Go module and pull in the dependencies:

mkdir my-exporter && cd my-exporter
go mod init example.com/my-exporter
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

Registering Metrics

Create main.go to declare and register your metrics with Prometheus's default registry. This ensures they appear in the output even before the first observation.

package main

import (
	"log"
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	jobsProcessed = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "worker_jobs_processed_total",
			Help: "Total number of jobs processed, partitioned by status.",
		},
		[]string{"status"},
	)
	queueDepth = prometheus.NewGauge(
		prometheus.GaugeOpts{
			Name: "worker_queue_depth",
			Help: "Current number of jobs waiting in the queue.",
		},
	)
	jobDuration = prometheus.NewHistogram(
		prometheus.HistogramOpts{
			Name:    "worker_job_duration_seconds",
			Help:    "Time spent processing a single job.",
			Buckets: prometheus.DefBuckets,
		},
	)
)

func init() {
	prometheus.MustRegister(jobsProcessed, queueDepth, jobDuration)
}

Collecting Real Values

Implement a collection loop (e.g., a goroutine) that periodically reads from your data source and updates the registered metrics. The polling interval should be shorter than Prometheus's scrape interval (typically 15 seconds) to ensure fresh data.

import (
	"math/rand"
	"time"
)

func collectMetrics() {
	for {
		// Replace with real reads from your application
		depth := float64(rand.Intn(50))
		queueDepth.Set(depth)

		start := time.Now()
		time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond)
		jobDuration.Observe(time.Since(start).Seconds())
		jobsProcessed.WithLabelValues("success").Inc()

		time.Sleep(5 * time.Second)
	}
}

Exposing the Endpoint

In main, wire the collection loop and the HTTP handler together. It is good practice to expose a separate /healthz path for Kubernetes liveness probes.

func main() {
	go collectMetrics()
	http.Handle("/metrics", promhttp.Handler())
	http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
	})

	log.Println("Listening on :8080")
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatalf("server error: %v", err)
	}
}

Containerization

Use a multi-stage Docker build to keep the final image small and secure. The first stage compiles a statically linked binary; the second stage copies only that binary into a minimal base image.

FROM golang:1.21-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /exporter .

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /exporter /exporter
EXPOSE 8080
ENTRYPOINT ["/exporter"]

The distroless/static:nonroot image contains no shell, no package manager, and runs as a non-root user by default, satisfying most cluster security policies without extra configuration.

Deployment to Kubernetes

Deploy the exporter using two manifests: a Deployment for pod lifecycle management and a Service for a stable address.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: my-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: my-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: my-exporter
    spec:
      containers:
      - name: exporter
        image: <registry>/my-exporter:v1.0.0
        ports:
        - name: metrics
          containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          requests:
            cpu: 50m
            memory: 32Mi
          limits:
            cpu: 100m
            memory: 64Mi
---
apiVersion: v1
kind: Service
metadata:
  name: my-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: my-exporter
spec:
  selector:
    app.kubernetes.io/name: my-exporter
  ports:
  - name: metrics
    port: 8080
    targetPort: metrics

Configuring Prometheus

How you configure scraping depends on your Prometheus installation.

Option 1: Prometheus Operator (ServiceMonitor)

If using the Prometheus Operator or kube-prometheus-stack, create a ServiceMonitor. Ensure the release label matches your Prometheus resource.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-exporter
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: my-exporter
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

Option 2: Annotation-based Discovery

For annotation-based setups, add these annotations to your Pod template:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"
  prometheus.io/path: "/metrics"

Verification

Port-forward to the Prometheus service and check the targets page:

kubectl port-forward svc/prometheus-operated 9090 -n monitoring

Navigate to http://localhost:9090/targets. The my-exporter target should show UP. Run a query in the expression browser to confirm data flow:

rate(worker_jobs_processed_total{status="success"}[2m])

Next Steps: Metrics Adapter

A working exporter feeds Prometheus, but to scale on these custom metrics, you need a metrics adapter. The Prometheus Adapter registers your custom metrics with the Kubernetes Custom Metrics API, allowing the HorizontalPodAutoscaler to reference them directly (e.g., worker_queue_depth). For a walkthrough of that setup, see the official documentation on Autoscaling on multiple metrics and custom metrics.