Skip to main content

Workflows & Versions

A workflow is a named container that owns everything related to a pipeline: its versions, configurations, and runs. Creating a workflow is a lightweight, idempotent operation — you give it an identifier and an optional display name.

// PUT .../workflows/ml-training-pipeline
{ "name": "ML Training Pipeline" }

Workflow Versions​

A workflow version captures the task DAG (the ordered set of steps) at a point in time. Each version belongs to a parent workflow and is identified by a versionId. Dots are allowed, so semantic-version-style values such as 1.0.0 are valid.

Versioning is the mechanism that lets you evolve a pipeline safely: an older version keeps running in production while you iterate on a newer one in test. Versions are what get assigned to stages and executed as runs.

Defining Tasks​

A version is a list of tasks. Each task references a workflow template via templateRef and supplies values through a flat arguments map. Tasks declare dependencies on other tasks by name, forming a DAG:

// PUT .../workflows/ml-training-pipeline/versions/1.0.0
{
  "tasks": [
    {
      "name": "fetch-git-code",
      "templateRef": { "name": "fetch-git-code", "version": "v1.0.0" },
      "arguments": {
        "git-url": "git@bmw.ghe.com:example/example-repo.git",
        "git-branch": "main"
      },
      "dependencies": []
    },
    {
      "name": "ray-training",
      "templateRef": { "name": "submit-ray-job", "version": "v1.0.0" },
      "arguments": {
        "entrypoint": "code/train.py",
        "output-bucket": "s3://${config.BUCKET}/output",
        "worker-replicas": "${config.RAY_REPLICAS}"
      },
      "dependencies": ["fetch-git-code"]
    }
  ]
}
Template versions

The templateRef.version values above are pinned to a full vMAJOR.MINOR.PATCH (v1.0.0) — an immutable pin, chosen here for reproducible runs. Task-template versions are always v-prefixed (a bare 1.0.0 is rejected). Use the floating alias v1 instead to always resolve the latest v1.x.x release. See Workflow Templates. The 1.0.0 in the request path above is the workflow version ID — a separate identifier, unaffected by this.

Arguments are a flat map

arguments is a flat string-to-string map ({"key": "value"}). The older {"parameters": [{"name": ..., "value": ...}]} shape is no longer used.

Referencing Configuration Values​

Argument values may embed ${config.KEY} placeholders (as with output-bucket and worker-replicas above). These are resolved at run time from the effective configuration, so the same version can run across every stage with different parameter values. Versions with no ${config.*} references behave exactly as before.

Validating a Version​

Before persisting, you can validate a task DAG (for example, to catch a dependency that points at a non-existent task) without saving it. Validation is informational — it reports a valid flag and a list of errors.

Deleting a Version​

Deleting a version also removes any stage assignment that references it. A version cannot be deleted while it has active (non-terminal) runs.

Next Steps​