Open Source Projects · Sub-chapter 7 · 4 min read

Kubeflow

Sub-chapter 7 of Open Source Projects · MLOps on Kubernetes, without writing Kubernetes

Kubeflow is the answer to a specific question: how do data scientists run distributed training, hyperparameter sweeps, and model serving on Kubernetes without learning Kubernetes? Originally a Google project, it grew into the canonical MLOps platform for the cloud-native world and joined CNCF as an Incubating project in 2023.

For an engineer who actually likes Kubernetes (or wants to), Kubeflow is a beautifully structured suite of CRDs and operators to learn from.

TL;DR

What - Open-source toolkit that makes deploying, scaling, and managing ML workflows on Kubernetes simple and portable.
License - Apache 2.0.
Governance - CNCF Incubating since July 25, 2023; previously a Google-led steering committee.
Origin - Started at Google in late 2017 as a way to run TensorFlow on Kubernetes.
Components - Pipelines, Notebooks, Training, Katib (HPO), KServe, Central Dashboard.

Why it exists

In 2017, Kubernetes was the platform-of-choice for new infrastructure, but the ML world hadn't caught up. Distributed training, Jupyter notebooks, hyperparameter tuning, and model serving each had their own bespoke setups. Data scientists were either learning Kubernetes plumbing (slow) or asking platform teams to provision specific tooling (slower).

Kubeflow started as Google's effort to solve this for TensorFlow and grew into a multi-framework MLOps platform.

In July 2023, after years under a Google-led steering committee, Kubeflow was accepted as a CNCF Incubating project. That move turned it from a "Google project" into a true community effort with maintainers from Red Hat, IBM, Bloomberg, Apple, NVIDIA, and others.

Architecture

Kubeflow is a suite of loosely coupled components, each running as Kubernetes CRDs/operators. You install only the pieces you need.

text

                    Central Dashboard
                          │
       ┌──────────┬───────┼───────┬───────────┐
       ▼          ▼       ▼       ▼           ▼
   Pipelines  Notebooks  Training Katib    KServe
    (KFP)     (Jupyter)  Operator  (HPO)  (serving)

Pipelines (KFP) - DAG-based workflow engine for ML. Author pipelines in Python; KFP compiles them to Argo Workflows.
Notebooks - managed Jupyter on the cluster, multi-tenant per Profile.
Training Operator / Trainer - distributed training jobs for PyTorch, TF, JAX, MPI.
Katib - hyperparameter tuning and neural architecture search.
KServe - model serving (originally KFServing; now its own CNCF Incubating project, often deployed alongside Kubeflow).
Central Dashboard & Profiles - multi-tenant UI; Profile = namespace + permissions.

You install the pieces you need via the kubeflow/manifests repo, or each standalone.

License and governance

License: Apache 2.0.
Foundation: CNCF (Linux Foundation), Incubating since Jul 25, 2023.
Steering Committee elected from active maintainers.

Install and hello world

You'll want a local Kubernetes cluster first. kind is the easiest:

bash

kind create cluster --name kubeflow
git clone https://github.com/kubeflow/manifests.git
cd manifests && git checkout v1.11-branch

# Apply manifests; the loop handles startup ordering.
while ! kustomize build example | kubectl apply -f -; do
  echo "retrying"; sleep 20
done

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

Open http://localhost:8080.

For something lighter, install a single component (e.g. KFP standalone) instead of the full platform:

bash

# KFP standalone
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=2.4.0"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=2.4.0"
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8888:80
# UI at http://localhost:8888

A minimal Python pipeline:

python

from kfp import dsl, compiler

@dsl.component
def say_hello(name: str) -> str:
    return f"hello {name}"

@dsl.pipeline(name="hello")
def hello_pipeline(name: str = "welzin"):
    say_hello(name=name)

compiler.Compiler().compile(hello_pipeline, "hello.yaml")
# Upload hello.yaml in the KFP UI to run it.

How to contribute

GitHub org: github.com/kubeflow - many repos by component.
Contributing guide: kubeflow.org/docs/about/contributing
Community page: kubeflow.org/docs/about/community
DCO - sign off commits with git commit -s.
Working Groups: AutoML, Pipelines, Training, Notebooks, Serving, Manifests, Security. Pick one.
Slack: CNCF Slack slack.cncf.io, channel #kubeflow-contributors.
Mailing list: groups.google.com/g/kubeflow-discuss.
Good first issue label is widely used across repos.

Where to start:

Docs / website - easiest entry-point; the kubeflow/website repo welcomes docs PRs.
Manifests - Kustomize/YAML; great for engineers who like infra.
Pipelines - Python + Kubernetes; large surface area, many issues.
KServe - if your interest is serving / inference (it's its own CNCF Incubating project now).

Hands-on Checkpoints

Spin up a kind cluster + KFP standalone. Run the "hello" pipeline above.
Author a 3-step pipeline (data prep → train → eval). Run it. Inspect the artifacts.
Read one Kubeflow Working Group's last 4 meeting notes. Pick the WG that matches your interest.
Find one good first issue in kubeflow/website or kubeflow/manifests. Open a PR.
Compare Kubeflow Pipelines and Argo Workflows. Note where they overlap and where they don't.