Agentic AI & LLM Applications
What You Will Build
This cookbook is a production-focused path for teams that need to ship an enterprise agent quickly:
- A CAIP-managed agent runtime with tool/function calling
- Unified model access through LLM API (self-hosted or managed models)
- Retrieval-augmented generation (RAG) using vector stores
- Persistent conversations (threads/messages)
- End-to-end observability with Langfuse
If you follow all six recipes in order, you can go from zero setup to a production-ready baseline in one implementation cycle, without building custom platform plumbing.
Ingredients
- CAIP Apps — managed hosting for the agent/chatbot application
- Agents Foundry — REST API + Python SDK (PydanticAI, LangChain, LangGraph) for building agent runtimes with tool/function calling
- LLM API — self-hosted or managed (Bedrock, Azure OpenAI, Alibaba Cloud) LLM backend
- Vector Store (Agents API) — semantic/keyword/hybrid search for RAG
- Langfuse — managed observability for traces, tool calls, latency, and cost
The Recipe
CAIP 2.0: Agentic AI
Agentic AI tooling is native to CAIP 2.0 — Managed Kubeflow (CAIP 1.5) predates agent-specific tooling and has no equivalent managed offering for agent hosting, LLM orchestration, vector-store/RAG, or agent-level observability. If you're on CAIP 1.5 today, this is a capability you'd need to build yourself (calling out to an LLM provider directly, with no managed hosting, vector store, or tracing built in) — it's a strong reason on its own to plan a move to CAIP 2.0. See Migration Support.
Hosting the Agent (CAIP Apps)
CAIP Apps provides a fully managed, scalable framework for deploying custom applications — from Streamlit dashboards to LLM-based chatbots — without operating the underlying infrastructure yourself. Create an application from a type and image, deploy it, and manage its full lifecycle (including autoscaling and monitoring) via API. See Apps: Getting Started.
For fully-built agent runtimes (not just app hosting), Agents Foundry combines a REST API with a lightweight Python SDK (supporting PydanticAI and LangChain) so ML scientists and developers can build, customize, and deploy agents with minimal engineering effort, including native tool/function calling — see Agents for the core concepts and lifecycle.
LLM Inference: Self-Hosted & Managed Models
Every agent needs an LLM backend. CAIP's LLM API gives you one unified interface (OpenAI-compatible endpoints) for:
- Self-hosted models — open-source models (chat, embeddings, vision, speech, image/video generation) running on CAIP-managed GPU infrastructure, zero additional cost, full data sovereignty. See Self-Hosted Models.
- Managed models — access to AWS Bedrock, Azure OpenAI, and Alibaba Cloud model catalogues through the same API, so you can pick the best model for the job without separate provider integrations. See LLM API and the Model Catalogue for the full list of available models, pricing, and rate limits.
RAG with CAIP (Vector Store)
To ground an agent's answers in your own documents rather than relying on the model's training data alone, CAIP's Agents API provides a built-in vector store with semantic, keyword, and hybrid search for Retrieval-Augmented Generation (RAG). See RAG & Vector Stores for the concepts, and the RAG Agent example for a worked implementation.
Agent Observability (Langfuse)
Once an agent is live, you need visibility into every LLM call, tool invocation, and decision — what was sent to the model, why the agent chose an action, how long each step took, and what it cost. CAIP provides a managed Langfuse instance (BMW WebEAM SSO) for exactly this, and it's already built into the Agents Foundry SDK. See Langfuse Observability.
If your agentic use case needs a hosted Model Context Protocol (MCP) server rather than (or in addition to) a CAIP Apps deployment, this can be served through ORBIT directly. Reach out via Support to discuss the right setup for your use case — dedicated CAIP-native tooling for this is being explored.
Step-by-Step Recipes
Building an agent breaks down into six concrete steps. Each has its own recipe below, in the order you'd typically follow them:
| Step | Recipe | What it covers |
|---|---|---|
| 1 | Create an Agent | Register agent metadata in the CAIP Portal, install the SDK, connect and run your first agent |
| 2 | Attach the CAIP LLM API | How agent model calls route through the LLM API, choosing self-hosted vs. managed models, calling the API directly |
| 3 | Add Conversation History | Threads and messages — persisting multi-turn conversations and passing context back to the agent |
| 4 | Add a Vector Store | Provisioning a vector store for RAG — embedding model, chunk size, and overlap choices |
| 5 | Create & Ingest Embeddings | Getting your documents into the vector store as embeddings |
| 6 | Search the Vector Store | Querying the vector store — vector, keyword, and hybrid search, and wiring it into an agent tool |
Production Outcome Checklist
Before go-live, confirm these outcomes are true:
- Agent configuration is versioned and reproducible (name/model/instructions/default parameters).
- Runtime uses unified auth (
CAIP_API_KEY) with least-privilege key management. - Thread persistence works across sessions and app restarts.
- RAG retrieval is tested for precision and fallback behavior.
- Langfuse traces include prompts, tool calls, latency, and token usage.
- Rollout uses a non-production environment first, then production with monitoring gates.
Related Recipes
- Need a model trained or fine-tuned first? See Model Training & Model Registry.
- Serving a classic ML model instead of building an agent? See Model & LLM Inference.