Skip to main content

Recipe: Attach the CAIP LLM API

The Problem​

"Our agent needs an LLM backend, but we don't want to manage separate credentials and SDKs per provider (OpenAI, Anthropic, Bedrock...), and we'd like the option to point at a self-hosted open-source model instead of paying per-token — all through one interface, with the same authentication everywhere."

Ingredients​

  • A unified CAIP API Key (CAIP_API_KEY)
  • Either the Agents SDK (model attached automatically), or the OpenAI-compatible LLM API directly (for non-agent code)

The Recipe​

1. Get a CAIP API key​

Obtain a CAIP API key through the Self-Service Portal or by raising a Service Request of type 'Request API Key'.

2. Path A — Let the Agents SDK attach it for you (recommended for agents)​

If you're building with the Agents SDK, the LLM API is already attached: the model/provider chosen when the agent was created in the Portal determines which model is called, and every agent.run() / agent.stream() call is routed through the LLM API's /v1/chat/completions endpoint using your CAIP_API_KEY. You don't call the LLM API yourself - the SDK handles authentication, retries, and endpoint selection.

.env
CAIP_API_KEY=your_actual_api_key_here
CAIP_REGION=ROW
# Optional: development | test | int | production
# CAIP_ENV=development

To change models without touching agent code, override at runtime:

agent = client.create_agent(
    framework="pydantic_ai",
    agent_id="your-agent-id",
    model_name_override="gpt-4-turbo",  # overrides the Portal setting
)

3. Path B — Call the LLM API directly (for scripts, or apps without the Agents SDK)​

Use any OpenAI-compatible SDK by pointing base_url at the CAIP endpoint:

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.api.caip.bmw.cloud/v1",  # use bmwchina.cloud for CN routing
    api_key="{CAIP_API_KEY}",
)

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}],
)

Or with curl:

curl -X POST https://llm.api.caip.bmw.cloud/v1/chat/completions \
    -H 'Authorization: Bearer {CAIP_API_KEY}' \
  -H 'Content-Type: application/json' \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello!"}]}'

4. Choose self-hosted vs. managed​

  • Self-hosted models — open-source models running on CAIP GPU infrastructure, zero additional token cost, full data sovereignty. See Self-Hosted Models.
  • Managed models — AWS Bedrock, Azure OpenAI, and Alibaba Cloud catalogues through the same API/key. See the Model Catalogue.

Both are referenced the same way — by model name — whether set in the agent's Portal configuration or passed directly to the LLM API.

Enterprise Runtime Pattern​

  • Default to SDK-managed model calls for agent workloads.
  • Keep direct LLM API calls for sidecar workflows (batch summarization, offline evaluation, migration scripts).
  • Use environment-specific keys and monitor token/latency trends in Langfuse before promoting model changes.

Validation Checklist​

  • CAIP_API_KEY works for both an agent call and a direct LLM API call.
  • CAIP_REGION and CAIP_ENV resolve to expected endpoints.
  • Chosen model is available in your target environment and region.
  • Retry behavior is verified for transient 429 and 5xx failures.