Self-Hosted Models

Overview

CAIP now offers self-hosted model options across nearly every endpoint modality. These models are open-source, run directly on CAIP-managed GPU infrastructure, and are accessible through the same standard LLMAPI endpoints you already use — just swap the model parameter.

Early Access

Self-hosted models are currently in early access / beta. Core inference works reliably, but some features and parameters may not yet be fully supported. See Known Limitations for details.

Why Self-Hosted?

Zero Additional Cost

These models run on GPU infrastructure that is already provisioned and available within the CAIP cluster. There is no per-token charge or consumption-based billing for self-hosted models at this time.

Runs Inside the CAIP Cluster

All self-hosted models run directly within the CAIP Kubernetes cluster and never leave our VPC. Traffic stays entirely within our network boundary — no external API calls, no third-party processing, no data egress. This is the strongest isolation level we can offer.

Full Data Sovereignty

Because inference happens on-cluster, your data never touches any external provider. Unlike managed models (Azure OpenAI, AWS Bedrock, Alibaba Cloud), self-hosted models process all requests entirely within CAIP-controlled infrastructure.

Open-Source Transparency

All self-hosted models are open-weight, auditable, and free from vendor lock-in. You can inspect the model architecture, weights, and licensing independently.

Available Models

All self-hosted models are accessible through the same standard LLMAPI endpoints. Simply use the model name in your request body — routing is handled transparently via LiteLLM.

Modality	Model Name	Underlying Model	Description
Chat Completions	qwen3-chat	Qwen/Qwen3-4B	Lightweight, fast chat model for general-purpose conversation
Chat / Coding (agentic)	qwen3.6-35b-a3b	Qwen/Qwen3.6-35B-A3B (FP8)	Larger MoE model for coding and agentic workflows — supports tool calling, vision, and a 128K context window
Embeddings	qwen3-embedding	Qwen/Qwen3-Embedding-0.6B	Text embeddings for search, RAG, and similarity
Speech-to-Text	qwen3-asr	Qwen/Qwen3-ASR-1.7B	Audio transcription and speech-to-text
Image Generation	flux-image	FLUX.2-klein-4B	Text-to-image generation
Video Generation	wan-video	Wan2.2-TI2V-5B	Text/image-to-video generation

Coding & Agentic Tooling

The qwen3.6-35b-a3b model is tuned for coding and agentic tool use. Unlike qwen3-chat, it supports tool / function calling, vision, and a 128K-token context window — making it suitable as a backend for terminal coding agents such as GitHub Copilot CLI and opencode. A reasoning variant, qwen3.6-35b-a3b-think, enables extended thinking. See Coding with the LLM API for step-by-step integration guides, or Image Input (Vision) for direct VLM request examples.

Known Limitations

Since self-hosted models are in early access, please be aware of the following:

• Throughput is currently limited — The self-hosted models share a fixed GPU allocation. Under high concurrency you may experience slower response times or queuing. We are actively working on scaling the infrastructure and this will be improved in future updates.
• Tool calling / function calling is not supported on the lightweight qwen3-chat model. Requests that include tools will be silently routed to a fallback model (e.g. gpt-4o-mini) instead of the self-hosted model. Check the model field in the response to verify which model processed your request. (The qwen3.6-35b-a3b coding model does support tool calling — see Coding with the LLM API.)
• Image input (vision) is not available on the qwen3-chat model. Requests with image content will be silently routed to a fallback model. Check the model field in the response. (The qwen3.6-35b-a3b model does support vision.)
• Some request parameters may not be supported or may behave differently compared to the managed OpenAI/Anthropic endpoints. Not all parameters from the OpenAI API specification are guaranteed to work.
• Model capabilities — Smaller open-source models (e.g. qwen3-chat at 4B parameters) will not match the quality of GPT-4o or Claude Sonnet 4.6 for complex reasoning tasks. Choose the right tool for the job.

Supported Features

The following features have been tested and work directly on the self-hosted qwen3-chat model:

Feature	Status	Notes
Basic chat completions	Supported	Standard message format works as expected
Streaming	Supported	SSE chunks with stream: true
JSON mode	Supported	response_format: {"type": "json_object"} works
Tool calling	Not supported	Silently falls back to managed model
Image/vision input	Not supported	Silently falls back to managed model

Silent Fallback Behavior

When using unsupported features (tool calling, vision), the API returns HTTP 200 but routes the request to a managed model instead of the self-hosted model. Always check the model field in the response to confirm which model processed your request.

We are actively working to close these gaps and increase capacity.

Coding model capabilities differ

The feature matrix above applies to qwen3-chat. The qwen3.6-35b-a3b coding model has an expanded capability set — it supports tool calling and vision natively (no silent fallback). See Coding with the LLM API.

Getting Started

Base URL

Self-hosted models use the same LLMAPI base URL as all other models:

RoW:

• PROD: https://llm.api.caip.bmw.cloud

China:

• PROD: https://llm.api.caip.bmwchina.cloud

Authentication

Authentication is identical to the managed models. See the LLM API Authentication docs for details.

Authorization: Bearer <your_api_key>

Usage Examples

No endpoint changes are required. Use the standard LLMAPI endpoints with the self-hosted model name in the model field.

Chat Completions (qwen3-chat)

Thinking Tokens

qwen3-chat includes reasoning/thinking tokens in its responses by default (wrapped in <think>...</think> tags). The actual answer follows after the thinking block. You may want to strip these in your application or use enable_thinking: false in the request body via extra_body to disable them.

curl -X POST \
  'https://llm.api.caip.bmw.cloud/v1/chat/completions' \
  -H 'Authorization: Bearer {Your_apikey}' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "qwen3-chat",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is 1+1?"}
  ]
}'

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.api.caip.bmw.cloud/v1",
    api_key="{Your_apikey}"
)

response = client.chat.completions.create(
    model="qwen3-chat",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is 1+1?"}
    ]
)
print(response.choices[0].message.content)

Embeddings (qwen3-embedding)

curl -X POST \
  'https://llm.api.caip.bmw.cloud/v1/embeddings' \
  -H 'Authorization: Bearer {Your_apikey}' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "qwen3-embedding",
  "input": "Introduction to self-hosted embeddings"
}'

response = client.embeddings.create(
    model="qwen3-embedding",
    input="Introduction to self-hosted embeddings"
)
print(f"Embedding dimension: {len(response.data[0].embedding)}")

Speech-to-Text (qwen3-asr)

curl -X POST \
  'https://llm.api.caip.bmw.cloud/v1/audio/transcriptions' \
  -H 'Authorization: Bearer {Your_apikey}' \
  -H 'Content-Type: multipart/form-data' \
  -F file="@/path/to/audio.mp3" \
  -F model="qwen3-asr"

audio_file = open("/path/to/audio.mp3", "rb")
transcription = client.audio.transcriptions.create(
    model="qwen3-asr",
    file=audio_file
)
print(transcription.text)

Image Generation (flux-image)

curl -X POST \
  'https://llm.api.caip.bmw.cloud/v1/images/generations' \
  -H 'Authorization: Bearer {Your_apikey}' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "flux-image",
  "prompt": "A futuristic BMW driving on Mars",
  "n": 1
}'

import base64

response = client.images.generate(
    model="flux-image",
    prompt="A futuristic BMW driving on Mars",
    n=1
)

image_data = response.data[0].b64_json
image_bytes = base64.b64decode(image_data)
with open("generated.png", "wb") as f:
    f.write(image_bytes)

Video Generation (wan-video)

Video generation is asynchronous. You submit a generation request, poll for completion, then download the result.

Step 1 — Create video:

curl -X POST \
  'https://llm.api.caip.bmw.cloud/v1/videos' \
  -H 'Authorization: Bearer {Your_apikey}' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "wan-video",
  "prompt": "A futuristic BMW driving through a neon-lit city at night"
}'

Response:

{
  "id": "video_abc123...",
  "object": "video",
  "status": "queued",
  "created_at": 1781282201,
  "model": "wan-video"
}

Step 2 — Poll status:

curl 'https://llm.api.caip.bmw.cloud/v1/videos/{video_id}' \
  -H 'Authorization: Bearer {Your_apikey}'

Wait until status changes to "completed".

Step 3 — Download content:

curl -o output.mp4 \
  'https://llm.api.caip.bmw.cloud/v1/videos/{video_id}/content' \
  -H 'Authorization: Bearer {Your_apikey}'

Video Generation Timeout

Video generation can take significantly longer than other modalities. Ensure your client timeout is set appropriately when polling.

Request a Custom Self-Hosted Model

Coming Soon

The ability to request custom self-hosted models for your team is currently being developed. At this stage, if you would like a specific model to be hosted on CAIP infrastructure, please reach out to the CAIP team and we will evaluate and integrate the model on your behalf.

To request a custom self-hosted model, submit a ticket here: Connected AI: Self Hosting Model.

Disclaimer

• Data Responsibility: The feature team is fully responsible for ensuring that any data used for model inference complies with local privacy and data protection regulations (e.g., PIA).
• Model Provenance: The use case owner must ensure all licensing, security scanning, and compliance checks are complete before requesting deployment.
• Security Boundary: Self-hosted models run in isolated namespaces within the CAIP cluster.
• Resource Limits: Default limits on GPU usage, storage, and replicas apply. Specify any custom hardware or scaling needs clearly in your request.
• Support Scope: The CAIP team does not provide debugging support for model logic or vendor-specific runtime issues.
• Monitoring and Logging: CAIP provides infra-level metrics (e.g., resource usage, availability) only.

Request Template

Section
(1) Incident Information Mandatory fields: - Caller: (requestor name) - Service offering: Connected AI platform - Assignment group: FT_Connected AI Platform-3rd
(2) Short description Domain_Use Case Name_Service Request for Self Hosting Model Example: IPA_Custom LLM_Service Request for Self Hosting Model
(3) Description Background: As a CAIP customer, I would like to self-host "Model Name" on CAIP for "Domain + Use Case Name" __________________ Model Information: - Model Name: (e.g. Qwen/Qwen3-8B) - Model Size: (parameter count, e.g. 8B) - Namespace: - Region: - Desired Modality: (chat, embeddings, STT, image, video) - Max Replicas: (optional) _______________________________ Additional Information (optional) _______________________________ Contact Information <<< enter here >>> <contact, tel.>

Observability

TBD: This section will be completed with detailed observability information.

• Infrastructure metrics (CPU, Memory, GPU utilization)
• Request latency and throughput
• Error rates and response codes

Troubleshooting and Support

If you encounter any issues, feel free to contact the CAIP team or raise tickets to the assignment groups below:

• FT_orbit-2nd — for RoW
• FT_orbit-cn-2nd — for CN