Skip to main content

CAIP Workflows Architecture

Architecture Overview

CAIP Workflows is a scalable platform for orchestrating complex machine-learning and data-processing pipelines on EKS. It uses Argo Workflows as the orchestration engine, KubeRay for distributed ML training, and cloud services for storage and persistence. Workflow definitions live in a database — the single source of truth — and are translated into Argo Workflow specifications only when a run is created.

The core components of the architecture include:

  1. CAIP Workflows API: A FastAPI-based RESTful API that is the primary interface for defining and managing workflows. It handles request validation, workflow versioning, stage promotion, and configuration resolution, then translates a version into an Argo Workflow specification at run time.

  2. Argo Workflows: The orchestration engine that executes workflows on Kubernetes. It manages the lifecycle of execution, including scheduling, monitoring, and retry logic.

  3. KubeRay Integration: For workflows that require distributed ML training, KubeRay manages Ray clusters on Kubernetes so training jobs can scale across multiple nodes.

  4. Shared Storage Layer: A shared storage solution (e.g., S3 / PVC) that provides a common workspace. Different stages of a workflow share data seamlessly, and intermediate results are persisted.

  5. Database (MongoDB Atlas): The source of truth for workflow definitions, versions, stage assignments, configurations, and execution metadata. This is what enables versioning and staging without duplicating Kubernetes resources across clusters.

Architecture Diagram​

CAIP Workflows Architecture

Core Concepts​

The API is organized around a small set of resources that together describe what runs, where, and with which parameters:

ConceptDescription
WorkflowA named container that owns versions, configurations, and runs.
Workflow VersionThe task DAG captured at a point in time. Versions are immutable definitions identified by a versionId (e.g. 1.0.0). See Workflows & Versions.
Stage AssignmentBinds a version (and optionally a configuration) to an environment: test, int, e2e, or prod. See Stages.
ConfigurationA named key/value map referenced from task arguments via ${config.KEY}, letting one version run across every stage. See Configurations.
RunA single execution of a version in a stage, backed by an Argo Workflow. See Runs.
TriggerStarts runs automatically — on a webhook call or a cron schedule — using the stage-bound version and configuration. See Triggers.

Lifecycle of a Workflow​

A typical end-to-end flow:

  1. Create a workflow — a container for everything below.
  2. Push a version — define the task DAG. Task arguments may reference configuration values with ${config.KEY}.
  3. Create configurations — e.g. dev-config and prod-config with environment-specific values.
  4. Assign to stages — bind the version and the matching configuration to each stage.
  5. Trigger a run — the API resolves all ${config.KEY} references, submits the workflow to Argo, and snapshots the exact values used for audit.

Only the run step interacts with Argo Workflows. Every step before it interacts only with the database, which remains the single source of truth throughout the lifecycle.

Why a Database-Centric Design​

  • Versioning: Argo has no native versioning; MongoDB tracks every workflow definition and configuration over time.
  • Staging & multi-region: Centralized metadata enables consistent execution across stages and regions without copying resources into each cluster.
  • Auditability: Because configurations are mutable, every run stores a snapshot of the resolved values it actually used.
  • Backward compatibility: Versions without ${config.*} references behave exactly as before — configurations are entirely optional.

For endpoint-level details, see the CAIP Workflows API Reference.