Kubeflow: ML Pipelines on Kubernetes

By Khimananda Oli | Last reviewed: August 2026

Moving machine learning from experimental notebooks to reliable production systems remains one of the hardest challenges in modern engineering. Kubeflow: ML Pipelines on Kubernetes solves this by treating ML workflows as first-class cloud-native applications rather than fragile scripts. It provides a portable, scalable orchestration layer that integrates directly with your existing infrastructure, enabling reproducible training runs and automated deployments. If you are already operating clusters using patterns from our MLOps from notebook to production guide, Kubeflow is the logical next step to enforce standardization.

What is Kubeflow: ML Pipelines on Kubernetes and why use it?

Kubeflow Pipelines (KFP) is the workflow orchestration engine at the heart of the Kubeflow ecosystem. While Kubeflow itself is a collection of tools for notebooks, training operators, and model serving, KFP specifically addresses the reproducibility crisis in machine learning. In practice, most teams start with Jupyter notebooks that work locally but fail when handed to operations because they lack dependency isolation, versioned parameters, or defined resource requests. KFP forces you to containerize every step, making the pipeline portable across any conformant Kubernetes cluster.

The primary value proposition is lineage and auditability. Every run generates a complete record of inputs, outputs, container images, and execution logs stored in ML Metadata (MLMD). For organizations pursuing SOC 2 or ISO 27001 compliance, this automated evidence collection is invaluable. You can trace exactly which dataset version produced a specific model artifact without digging through Slack history or local filesystems. This aligns with the observability principles discussed in our monitoring ML models in production article, extending visibility from runtime metrics back to training provenance.

Beyond compliance, KFP enables genuine collaboration. Data scientists define logic in Python; platform engineers manage the underlying infrastructure; and auditors verify the process—all through the same interface. The decoupling of pipeline definition from execution means you can develop locally on Kind or Minikube and deploy to EKS, GKE, or AKS without rewriting code. This portability prevents vendor lock-in, a critical consideration for Nepali tech companies and global startups alike who need flexibility in cloud provider selection.

How do you install Kubeflow Pipelines on a production cluster?

Installation is where many teams stumble. Avoid the "full Kubeflow" distribution unless you need every component (Notebooks, Katib, KServe). For pure pipeline orchestration, the standalone KFP installation is lighter, easier to upgrade, and less prone to CRD conflicts. As of 2026, the recommended approach uses Kustomize with environment-specific overlays.

Prerequisites and Cluster Sizing

• Kubernetes Version: v1.28+ required for current KFP v2.x releases.
• Storage: A default StorageClass must be configured. Refer to Kubernetes Persistent Volumes and storage if your cluster lacks dynamic provisioning.
• Resources: Minimum 4 CPU / 8GB RAM for control plane components. Add node capacity for actual workload execution.
• Container Registry: Accessible registry (ECR, GAR, Harbor) for pushing pipeline step images.

Standalone Installation Commands

# Clone the official manifests
git clone https://github.com/kubeflow/pipelines.git
cd pipelines/manifests/kustomize

# Configure environment (e.g., AWS EKS with S3 backend)
export PIPELINE_VERSION=2.3.0
kubectl apply -k cluster-scoped-resources
kubectl wait --for=condition=established crd/applications.app.k8s.io

# Apply namespaced resources
kubectl apply -k env/platform-agnostic-prow

# Verify pods are running
kubectl get pods -n kubeflow

A common mistake is skipping the CRD wait condition. Kustomize applies resources asynchronously; if the Application CRD isn't established before dependent resources are created, the installation fails silently. Always verify pod health before proceeding. For air-gapped environments common in government or regulated sectors, pre-pull all images to your internal registry and update the Kustomize patches accordingly.

How do you build reproducible pipelines with the KFP SDK?

KFP v2 uses a decorator-based Python DSL that compiles to IR YAML. Each function decorated with @dsl.component becomes a containerized step. The key discipline here is treating components as pure functions: explicit inputs, explicit outputs, no hidden side effects.

Defining Components and Pipelines

from kfp import dsl, compiler
from kfp.dsl import Dataset, Model, Output, Input

@dsl.component(base_image='python:3.11-slim')
def preprocess_data(raw_data: str, output_dataset: Output[Dataset]):
    """Pure preprocessing step with explicit I/O."""
    import pandas as pd
    df = pd.read_csv(raw_data)
    # Cleaning logic here
    df.to_parquet(output_dataset.path, index=False)

@dsl.component(base_image='tensorflow/tensorflow:2.16.1-gpu')
def train_model(dataset: Input[Dataset], model: Output[Model], epochs: int = 10):
    """Training step requesting GPU resources."""
    import tensorflow as tf
    # Load from dataset.path, train, save to model.path
    pass

@dsl.pipeline(name='fraud-detection-training')
def training_pipeline(data_uri: str, epochs: int = 20):
    prep_task = preprocess_data(raw_data=data_uri)
    train_task = train_model(
        dataset=prep_task.outputs['output_dataset'],
        epochs=epochs
    ).set_gpu_limit(1)

if __name__ == '__main__':
    compiler.Compiler().compile(training_pipeline, 'pipeline.yaml')

Notice the explicit typing (Input[Dataset], Output[Model]). This isn't just documentation—it enables the KFP runtime to handle artifact serialization, storage paths, and metadata registration automatically. Never hardcode S3 paths inside components; let the framework inject them. This ensures portability and makes local testing possible. For teams managing complex dependencies, refer to our Docker Buildx multi-platform builds guide to create optimized base images that reduce cold-start latency in pipeline steps.

How does Kubeflow compare to Airflow and SageMaker Pipelines?

Choosing an orchestrator depends on your team's skills, compliance needs, and cloud strategy. There is no universal best option—only trade-offs.

When to choose KFP: You run multi-cloud or hybrid, need strict data sovereignty (common in Nepal's fintech sector), or require deep ML lineage without vendor lock-in. When to choose Airflow: Your workflows are primarily ETL/data engineering with occasional ML, and your team lacks Kubernetes expertise. When to choose SageMaker: You are fully committed to AWS, want zero operational overhead, and accept the cost premium for managed convenience.

In my experience helping teams achieve SOC 2 compliance, KFP's self-hosted nature gives auditors confidence because you control the entire stack. Airflow often requires additional tooling for ML lineage, while SageMaker ties your compliance posture to AWS's certification scope. For organizations building AI adoption roadmaps, starting with KFP preserves future optionality.

How do you secure and optimize Kubeflow for production workloads?

Running KFP in production demands more than a successful install. Security, cost control, and reliability require deliberate configuration.

Security Hardening Checklist

1. Namespace Isolation: Deploy KFP in a dedicated namespace. Use NetworkPolicies to restrict egress from pipeline steps to only required endpoints (artifact store, model registry).
2. RBAC: Create per-team ServiceAccounts with minimal permissions. Never run pipeline steps with cluster-admin privileges.
3. Secrets Management: Integrate with External Secrets Operator or Vault. Pass credentials as mounted volumes, never environment variables. See Kubernetes secrets management done right for implementation patterns.
4. Image Policy: Enforce signed, scanned images via admission controllers (Kyverno/OPA). Block mutable tags like :latest.
5. Encryption: Enable TLS for intra-cluster communication and encrypt artifact storage at rest.

Cost Optimization Strategies

ML workloads can bankrupt a startup overnight. Implement these controls early:

• Spot/Preemptible Nodes: Configure KFP to schedule training jobs on spot instances. Handle interruptions gracefully with checkpointing.
• Resource Requests: Mandate CPU/memory requests in every component. Unbounded pods cause noisy neighbor issues and billing surprises.
• TTL Controller: Auto-delete completed workflow pods after 24 hours to reclaim etcd storage and node resources.
• Caching: Enable KFP's built-in caching for deterministic steps. Re-running identical preprocessing wastes money.

Monitoring and Observability

KFP exposes Prometheus metrics at /metrics on the API server and workflow controller. Track these golden signals:

• Pipeline Run Duration: P95 latency by pipeline name. Spikes indicate resource contention or upstream data issues.
• Step Failure Rate: Broken down by component. High failure rates in specific steps suggest flaky code or unstable dependencies.
• Cache Hit Ratio: Low ratios mean you're recomputing unnecessarily. Investigate non-deterministic inputs.
• Queue Depth: Pending workflows signal insufficient cluster capacity or scheduler bottlenecks.

Integrate these with your existing Prometheus and Grafana monitoring stack. Alert on failure rate thresholds, not individual failures—ML pipelines are inherently stochastic, and over-alerting causes fatigue.

Next Steps for Production ML Orchestration

Kubeflow: ML Pipelines on Kubernetes provides the foundation for mature, auditable MLOps—but it rewards preparation. Start with a single, well-defined pipeline (not your most complex one) to validate your installation, security model, and artifact storage. Document your component conventions early; consistency pays exponential dividends as your pipeline library grows. Remember that the goal isn't perfect tooling but reliable, reproducible outcomes that survive audits and team turnover.

If you're evaluating KFP for your organization or struggling with production hardening, I help teams design compliant, cost-effective ML platforms. Reach out to discuss your specific requirements—whether you're in Kathmandu or operating globally, we can build a system that scales safely.