Skip to main content

Model & LLM Inference

The Problem​

"We have a trained model artifact — say a fraud-detection classifier — and need to expose it behind a real-time endpoint with a latency budget (e.g. sub-200ms p99) that autoscales with traffic, while a separate nightly job needs to score millions of historical rows in batch against the same model. We also want to run inference against our own fine-tuned/open-source LLM instead of paying per-token to an external provider for every request, and we need this to work without us operating the serving infrastructure (autoscaling, GPU allocation, artifact retrieval) by hand."

Serving is where a model actually delivers business value, and requirements differ a lot: a fraud model needs sub-second real-time responses, a nightly scoring job just needs to process millions of rows efficiently, and an LLM-backed feature needs GPU-backed serving without vendor lock-in.

Ingredients​

  • CAIP Inference API (CAIP 2.0, recommended) — Live Inference (Ray Serve) and Batch Inference (Ray Data)
  • Self-Hosted LLMs / LLM API (CAIP 2.0, recommended) — GPU-backed LLM serving, zero extra token cost
  • or KServe (CAIP 1.5 alternative) — batch and live model serving on Managed Kubeflow
  • Your trained/registered model artifact

The Recipe​

CAIP 2.0 (Recommended): CAIP Inference API + Self-Hosted LLMs​

  • Classic ML — Inference API: a FastAPI-based interface that orchestrates both serving patterns for you — Live Inference via Ray Serve deployments (autoscaling, real-time), and Batch Inference via Ray Data jobs (large-scale scoring), with model artifacts/metadata stored in MLflow/S3/DynamoDB. See Inference: Getting Started and Inference Architecture.
  • LLM inference — Self-Hosted Models: run open-source (or your own fine-tuned) LLMs directly on CAIP GPU infrastructure — zero additional token cost, full data sovereignty (traffic never leaves the CAIP VPC), reachable through the same standard LLM API endpoints already in use elsewhere in your org. See Self-Hosted Models for the model catalogue, request process, and current (early-access) limitations.
Need a managed LLM instead?

If self-hosting isn't the right fit, CAIP's LLM API also gives you unified access to managed models from AWS Bedrock, Azure OpenAI, and Alibaba Cloud — see Agentic AI & LLM Applications.

CAIP 1.5 Alternative: Kubeflow Model Serving (KServe)​

On Managed Kubeflow, both serving patterns are available through KServe:

  • Batch Inference — load a registered model (e.g. from MLflow) in a Kubeflow Pipeline step, process S3 data in batches, and schedule recurring scoring jobs.
  • Live Inference — serve a model behind a KServe endpoint for real-time predictions.

See Model Serving for the full how-to on both patterns.