Live Inference
This is an end-to-end walkthrough guide on how to run and manage live inference jobs using the CAIP Inference API.
This example will cover how to:
- Create a new live inference application
- List live inference applications
- Delete a live inference application
- Make live inference prediction
- Check live inference status
You can download the most recent Bruno Library:
- mkdir caip-inference-api-collection
- cd caip-inference-api-collection
- git clone --filter=blob:none --sparse -b main bmw@bmw.ghe.com:connected-ai/caip-inference-api.git collection
Description of endpoints
1. Create new live inference application
To create a new live inference application, you can use the following endpoint:
- POST Create (/v1/spaces/
<space_id>/live)
This endpoint allows you to specify the configuration for your live inference application: namespace, stage, model to be used, transforms settings, and resource allocation. Once the application is created, you will receive an ID that can be used to manage and make predictions with the application.
To create a live inference application, you can use curl or any API client (i.e. Bruno) to send a POST request to the endpoint /v1/spaces/<space_id>/live with the necessary configuration in the request body (JSON Payload) defining the required parameters.
space_id: [Your Space ID] (you can use "usecase-a" for testing purposes)
The application will concatenate your input data, and will use the combination of namespace and stage to pull the model from the specified MLFlow workspace and also to match with the specific Ray instance of your workspace.
JSON payload example to Create New Live Inference Application
{
"namespace": "usecase-a",
"stage": "test",
"model": {
"type": "sklearn",
"provider": {
"name": "mlflow",
"config": {
"uri": "mlflow-artifacts:/<your model name>"
}
}
},
"transforms": {
"preprocessing": [],
"postprocessing": []
},
"infrastructure": {
"numReplicas": 1,
"numCpusPerReplica": 1,
"numGpusPerReplica": 0
}
}
Response example:
{
"id": "07580a52-2b77-4446-aad8-de3d0df40a05",
"namespace": "usecase-a-test",
"endpoint": "/v1/live/prediction/07580a52-2b77-4446-aad8-de3d0df40a05",
"model": {
"type": "sklearn",
"provider": {
"name": "mlflow",
"config": {
"uri": "mlflow-artifacts:/mlflow_wine_model"
}
}
},
"transforms": {
"preprocessing": [],
"postprocessing": []
},
"infrastructure": {
"numReplicas": 1,
"numCpusPerReplica": 1,
"numGpusPerReplica": 0
},
"createdAt": "2026-06-16 14:23:08.739137"
}
From this example you can extract the id to be used within the Delete and Predict endpoints, and also the endpoint to be used for making predictions.
2. List live inference applications
To list all live inference applications created, you can use the following endpoint:
- GET List (/v1/spaces/
<space_id>/live)
This endpoint will return a list of all live inference applications deployed within the specified spaceId, along with their details such as ID, namespace, model, transforms, status, and other relevant information.
space_id: [Your Space ID] (you can use "usecase-a" for testing purposes)
Example of response:
{
"applications": [
{
"id": "9410cd90-28a0-4e3d-9d72-0d1aac35f5cd",
"namespace": "ai-usecase-test",
"endpoint": "/v1/live/prediction/9410cd90-28a0-4e3d-9d72-0d1aac35f5cd",
"model": {
"type": "sklearn",
"provider": {
"name": "mlflow",
"config": {
"uri": "mlflow-artifacts:/mlflow_wine_model"
}
}
},
"transforms": {
"preprocessing": [],
"postprocessing": []
},
"infrastructure": {
"numReplicas": 1,
"numCpusPerReplica": 1,
"numGpusPerReplica": 0
},
"createdAt": "2026-06-11 16:45:02.917912"
}
]
}
3. Delete live inference application
To delete a live inference application, you can use the following endpoint:
- DELETE Delete (/v1/spaces/
<space_id>/live/<id>)
This endpoint allows you to delete a specific live inference application by providing its id, space_id.
Deleting a live inference application is irreversible, it will delete it in Ray Cluster and DynamoDB, make sure to use this endpoint with caution.
id: [ID of the live inference application you want to delete]
space_id: [Your Space ID] (you can use "usecase-a" for testing purposes)
Response example:
{
"success": true,
"message": ""
}
4. Make live inference prediction
To make a live inference prediction using a specific live inference application, you can use the following endpoint:
- POST Predict (/v1/spaces/
<space_id>/live/prediction/<id>)
This endpoint allows you to send input data to the specified live inference application and receive a prediction in response. You will need to provide the input data in the request body, and the application will process it and return the prediction result.
id: [ID of the live inference application you want to use for prediction] space_id: [Your Space ID] (you can use "usecase-a" for testing purposes)
Body Example for Make Live Inference Prediction
{
"data": "mpmZmZmZHUBmZmZmZmbmPwAAAAAAAAAAZmZmZmZm/j/b+X5qvHSzPWAAAAAAACZAAAAAAAAAQUCSXP5D+u3vPxSuR+F6FAxA7FG4HoXr4T/NzMzMzMwiQDMzMzMzMx9AKVyPwvUo7D8AA4AAAA4AAM3MzMzMzARASgwCK4cWuT8AAAAAAAA5QAAAAAAAwFBAveMUHcnl7z+amZmZmZkJQMP1KFyPwuU/mpmZmZmZI0AzMzMzMzMfQFK4HoXrUeg/exSuR+F6pD9mZmZmZmYCQFpk099Pjbc/AAAAAAAALkAAAAAAAABLQIGVQ4ts5+8/FK5H4XoUCkDNzMzMzMzkP5qZmZmZmSNA",
"shape": [3, 11],
"dtype": "float64"
}
5. Check live inference status
To check the status of a live inference application, you can use the following endpoint:
- GET Status (/v1/spaces/
<space_id>/live/status)
This endpoint will return the current status of the specified live inference application, indicating whether it's running, stopped, or in an error state. This information can be useful for monitoring the health and performance of your live inference applications.
id: [ID of the live inference application you want to check the status of]
space_id: [Your Space ID] (you can use "usecase-a" for testing purposes)
The response provides the current information in the RayCluster related to the live inference application, such as the status of the deployment and its replicas, which can be useful for troubleshooting and monitoring purposes.
Response example:
{
"applications": [
{
"id": "5be2bd19-d25b-4fd8-8055-2530405b1072",
"namespace": "usecase-a-test",
"status": "RUNNING",
"message": "",
"deployments": [
{
"name": "LiveInferenceApp",
"status": "HEALTHY",
"message": "",
"replicas": [
{
"replicaId": "x146dqje",
"state": "RUNNING"
}
]
}
]
},
{
"id": "3acbcc17-d96c-40ce-a5ab-c6ea6a139fd8",
"namespace": "usecase-a-test",
"status": "RUNNING",
"message": "",
"deployments": [
{
"name": "LiveInferenceApp",
"status": "HEALTHY",
"message": "",
"replicas": [
{
"replicaId": "0upjrfra",
"state": "RUNNING"
}
]
}
]
},
{
"id": "db948b8d-74d6-448c-9fc8-4f79a0daff15",
"namespace": "usecase-a-test",
"status": "RUNNING",
"message": "",
"deployments": [
{
"name": "LiveInferenceApp",
"status": "HEALTHY",
"message": "",
"replicas": [
{
"replicaId": "5lnwtvje",
"state": "RUNNING"
}
]
}
]
}
]
}