Skip to main content

Data Processing & Orchestration

The Problem​

"We have a PySpark script that a data engineer runs locally (spark-submit against a local/laptop Spark session or a personal EMR sandbox) against sample data. It joins several raw sources (vehicle telemetry, CDH exports, third-party feeds) — tens of GB to low-TB per run, sometimes millions of small JSON/Parquet files — deduplicates records, validates schemas, and computes aggregated features. Right now it only runs on one person's machine with hardcoded local file paths and credentials. We need the same logic to run reliably on a schedule (or when new data lands via webhook), against real S3 data, promoted through test → int → prod without hand-editing the script per environment, with retry/monitoring, and without us having to stand up or size our own Spark/EMR cluster."

This is one of the most common starting points for a team on CAIP: before any model can be trained or served, data has to be reliably processed at scale, and a script that "works on my machine" has to become a repeatable, versioned, multi-environment pipeline.

Ingredients​

  • CAIP Workflows (CAIP 2.0, recommended) — orchestration engine for containerized tasks, with configs, stages, and triggers
  • or AWS Glue / EMR + Kubeflow Pipelines (CAIP 1.5 alternative) — fully managed Spark, invoked as pipeline components
  • Your existing PySpark/ETL script, containerized (no rewrite needed)

The Recipe​

CAIP Workflows is the CAIP 2.0 orchestration engine: it runs your containerized task(s) as an Argo Workflow on EKS, with workflow definitions, versions, stage assignments, and configurations tracked centrally so nothing needs to be duplicated per environment. See Workflows Architecture.

Steps: taking your local Spark script to CAIP Workflows​

  1. Containerize, don't rewrite. Wrap your existing transform.py PySpark script and its dependencies in a container image (the UV package manager is supported for distributing Python dependencies). The script's logic doesn't need to change.
  2. Parameterize instead of hardcoding. Replace hardcoded local paths/credentials in the script's arguments with ${config.KEY} references, e.g. ${config.INPUT_PATH}, ${config.OUTPUT_BUCKET} — see Configurations.
  3. Push a workflow version — even a single-task DAG is a valid version. See Workflows & Versions.
  4. Create a configuration per environment (e.g. dev-config pointing at test buckets, prod-config pointing at production buckets) and bind each to its stage (test, int, e2e, prod) — see Stages. The same image and version now run unchanged everywhere.
  5. Trigger manually first via curl/the portal to validate output against real data, then attach a schedule (cron) or webhook trigger (e.g. fired when new data lands, or from your CI) so it runs automatically — see Triggers.
  6. Inspect runs — every run is tracked with logs and a snapshot of the exact configuration values used, so a 3am failure is debuggable via API/portal rather than someone's terminal history. See Runs.

Start with Workflows: Getting Started and the end-to-end API walkthrough; the example repository has runnable, forkable versions of every step above (01-hello-world, 04-configuration, 07-full-lifecycle).

CAIP 1.5 Alternative: AWS Glue / EMR + Kubeflow Pipelines​

If your team is on Managed Kubeflow, the same problem is solved with fully managed Spark services instead of bringing your own container:

  • AWS Glue — serverless, fully managed Spark, less setup, higher cost per job, less customizable.
  • AWS EMR — more configuration responsibility, more customizable and cost-efficient at scale.

Both integrate with the Glue Data Catalog and can be invoked as pre-built components inside a Kubeflow Pipeline, giving you the same "chain steps + recurring runs" capability as Workflows' stages/triggers. See Data Engineering: Glue vs EMR for a full comparison, and Kubeflow Pipelines for how ETL steps chain into training/eval/deployment.

Which one should I use?

Already on Managed Kubeflow and want a fully-managed Spark service (no Dockerfile of your own)? Use Glue/EMR + Kubeflow Pipelines. Want to keep full control over your own Spark image/dependencies, get webhook-driven triggers, or you're building fresh on CAIP 2.0? Use CAIP Workflows — see Migration Support if you need to move an existing Kubeflow pipeline over.