Introducing the Headlamp Kubeflow Plugin
Kubeflow packages AI/ML workloads as Kubernetes Custom Resource Definitions (CRDs), which gives operators the ability to manage them with standard Kubernetes tools. In practice, however, the dedicated ML dashboards hide the underlying Kubernetes layer, forcing operators to switch to kubectl for low‑level troubleshooting. The Headlamp Kubeflow plugin bridges that gap by surfacing Kubeflow CRDs inside Headlamp, a flexible, open‑source Kubernetes web UI.
Why a Separate Operator View Matters
Data scientists use purpose‑built dashboards to launch notebooks, pipelines, and experiments. Operators, on the other hand, need to answer questions such as:
- Is a notebook failing because of
ImagePullBackOff,OOMKilled, or a pending PersistentVolumeClaim? - Which Katib trials have completed and which one is currently the best?
- What is the status of distributed training jobs across namespaces?
The plugin reads directly from the Kubernetes API server, presenting pod conditions, failure reasons, and resource relationships without an intermediary ML service.
Supported Kubeflow Components
The plugin automatically discovers installed Kubeflow API groups and renders the following resources:
- Notebooks: Jupyter, VS Code, RStudio –
Notebook,Profile,PodDefault - Pipelines:
Pipeline,PipelineVersion,Run,RecurringRun,Experiment - Katib (hyper‑parameter tuning):
Experiment,Trial,Suggestion - Training Runs:
TrainJob,TrainingRuntime,ClusterTrainingRuntime - Spark:
SparkApplication,ScheduledSparkApplication
Key UI Views
Notebook detail view aggregates pod conditions, CPU/GPU requests, volume mounts, environment variables, sidecar containers, and node tolerations—information that would normally require several kubectl describe commands.
Katib view displays the tuning algorithm, search space, live status of each trial, and highlights the current best trial with its metric values and parameter assignments.
Pipeline view reads the underlying Kubernetes resources directly, allowing operators to inspect pipeline state even when the Kubeflow Pipelines backend service is unavailable. The detail view offers a side‑by‑side YAML diff of the latest and previous PipelineVersion specifications.
Visualizing Relationships
The plugin registers a Headlamp map source that renders supported resources as graph nodes. Edges are drawn based on .metadata.ownerReferences, giving operators a quick visual of how notebooks, experiments, runs, and other objects relate to one another.
Getting Started
Installation instructions are provided in the plugin’s README, including a lightweight “CRD‑only” evaluation path that works with a minimal Kubeflow installation. Because the plugin discovers API groups at runtime, it can be added to an existing modular Kubeflow deployment or used on a fresh cluster that only contains the necessary CRDs.
Broader Pattern for CRD‑Heavy Platforms
The Headlamp Kubeflow plugin demonstrates a reusable pattern: expose platform‑specific custom resources through a generic Kubernetes UI, letting operators stay within a single toolset while still accessing the detailed state of underlying workloads.
Licensing and Contribution
Headlamp is maintained by the Kubernetes SIG UI and released under the Apache 2.0 license. Contributions, issue reports, and pull requests are welcomed via the Headlamp plugins repository.