Skip to main content

Model Training & Model Registry

The Problem​

"We need to train a classic ML model — say a gradient-boosted tree or a CNN — with a hyperparameter sweep across dozens of runs, log parameters/metrics for each run so we can compare them, and once we pick a winner, register that exact model version somewhere a serving layer can retrieve it by URI (not a file someone emails around). Separately, another team wants to fine-tune an open-weight LLM (e.g. LoRA fine-tuning a 7B-parameter model) on domain-specific data, which needs multi-GPU distributed compute we don't want to provision and size ourselves — and once trained, that checkpoint needs to be reachable through the same LLM endpoint the rest of the org already calls, not a bespoke one-off deployment."

Training spans a wide spectrum on CAIP — from a data scientist iterating in a notebook, to a distributed multi-GPU fine-tuning job — and every trained model needs a clear path into a registry so it can be found, versioned, and promoted to serving.

Ingredients​

  • CAIP Workflows (CAIP 2.0, recommended) — containerized training/fine-tuning execution, with KubeRay for distributed, multi-GPU jobs
  • LLM Catalogue / Self-Hosted Models (CAIP 2.0, recommended) — registry + serving for fine-tuned LLM checkpoints
  • or Kubeflow Notebooks/Pipelines + GPU Compute + MLflow (CAIP 1.5 alternative) — self-service registry for classic ML models
  • Your training/fine-tuning script (PyTorch, Hugging Face Trainer, etc.)

The Recipe​

  • Training & fine-tuning execution: package your training/fine-tuning script (PyTorch, Hugging Face Trainer, etc.) as a containerized task and run it through CAIP Workflows. Workflows uses Argo Workflows for general task orchestration and KubeRay to spin up Ray clusters on Kubernetes specifically for the distributed, multi-node/multi-GPU parts of a training job — so you get distributed compute without operating the cluster yourself. See Workflows Architecture and Getting Started.
  • Model registry (LLMs): rather than standing up your own registry for a fine-tuned LLM checkpoint, request it be added to CAIP's self-hosted model catalogue — CAIP owns the registry and serving, and the model becomes reachable through the same LLM API endpoint the rest of the org already uses (just swap the model parameter). See Self-Hosted Models for the request process.
Classic ML registry on CAIP 2.0

CAIP 2.0 does not yet have a self-service model registry for arbitrary classic ML models (the LLM catalogue above is LLM-specific and onboarded on request). If you need a self-service registry today for a classic ML model, use the CAIP 1.5 / MLflow path below.

CAIP 1.5 Alternative: Kubeflow Notebooks/Pipelines + GPU Compute + MLflow​

  • Training & testing: prototype interactively in Jupyter/VS Code Server Notebooks, or run production training as a Kubeflow Pipeline step for repeatable, trackable runs. See Train and Test.
  • GPU-backed compute: request GPU node pools (e.g. 1+ NVIDIA GPU) directly on a notebook or pipeline step for training that needs acceleration. See GPU Compute.
  • Model registry: MLflow is CAIP's fully self-service experiment-tracking and model-registry tool on Managed Kubeflow — log every run's parameters/metrics, and register the winning model version so it can be retrieved by URI for serving. See Register a Model in MLflow.