Deploying and Serving Your Model with KServe

Introduction

Once your model is trained and registered in MLflow, you need to deploy it to make predictions available through an API endpoint. This section guides you through using KServe to deploy your model, expose the inference endpoint, and test the deployed service.

The deployment flow looks like this:

Client
  ↓
Traefik Ingress
  ↓
Istio Ingress Gateway
  ↓
KServe InferenceService
  ↓
Model Pod

Deploying Your Model via KServe

Understanding KServe Deployment

KServe automates the deployment and serving of machine learning models. Rather than manually creating Kubernetes manifests, you can programmatically generate KServe configuration files that define the API, resource limits, and scaling behavior for your model.

Step 1: Create the InferenceService Configuration

Create a new Python file in your components folder that generates the KServe InferenceService YAML.

Create inference_service.py

def get_inference_service(
    name: str,
    namespace: str,
    predictor_framework: str,
    storage_uri: str,
    predictor_runtime_version: str = "latest",
) -> str:
    return f"""
    apiVersion: "serving.kserve.io/v1beta1"
    kind: "InferenceService"
    metadata:
      name: {name}
      namespace: {namespace}
      annotations:
        "sidecar.istio.io/inject": "false"
      labels:
        inferenceType: traefik-intranet
    spec:
      predictor:
        serviceAccountName: default-editor
        {predictor_framework}:
          storageUri: {storage_uri}
          runtimeVersion: {predictor_runtime_version}
          resources:
            limits:
              memory: "1Gi"
              cpu: "500m"
            requests:
              memory: "100Mi"
              cpu: "100m"
"""

Step 2: Integrate the InferenceService into Your Pipeline

Add inference service to pipeline.py

Import the inference service function and pass it to the KServe operator in your pipeline.

from ml_pipeline.components.inference_service import get_inference_service

# ... rest of the pipeline

deploy_inference_service_step = kserve_op(
    action="apply",
    inferenceservice_yaml=get_inference_service(
        name=model_name,
        storage_uri=model_version_s3_path,
        predictor_framework="xgboost",
        namespace=profile,
    ),
)

Exposing the Inference Endpoint

Prerequisites

To expose your inference endpoint outside the cluster, you need direct access to your Kubernetes cluster through kubectl.

This requires setting up the ORBIT-USE CLI.

ORBIT-USE CLI Setup

Install and configure orbit-use CLI

Install the orbit-use CLI

Install the CLI using pipx (recommended):

pipx install orbit-use --pip-args="--extra-index-url https://packages.orbit.bmwgroup.net/artifactory/api/pypi/funmarkt-orbit-use/simple"

Or install it using pip:

pip3 install orbit-use --index-url https://packages.orbit.bmwgroup.net/artifactory/api/pypi/funmarkt-orbit-use/simple"

Generate Default Credentials

Execute the following command to authenticate and configure your environment for CAIP access.

The CLI also configures cloud provider access, allowing you to use tools such as aws with the configured profile.

orbit-use cloud <orbit-space> <dev|prod> <orbit-teamspace> --role caip-developer

For more details, refer to the official documentation:

https://docs.caip.bmw.cloud/managed-kubeflow/iam/?_highlight=orbit#logging-in-with-orbit-use-cli

Deploying to the Intranet

Your inference endpoint can be exposed internally through the corporate intranet, with optional public internet exposure if required.

We recommend starting with private intranet exposure before enabling public access.

Step 1: Deploy the InferenceService

Create an inferenceservice.yml file with your model deployment configuration.

Create inferenceservice.yml

⚠️ Important

For intranet access, your InferenceService must include the inferenceType: traefik-intranet label.

This label matches the Knative config-domain selector used for the Traefik private domain configuration.

Without this label, the service will not receive a private Traefik domain.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris-intranet
  namespace: mcaip-kflw
  annotations:
    sidecar.istio.io/inject: "false"
  labels:
    inferenceType: traefik-intranet
spec:
  predictor:
    serviceAccountName: default-editor
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://cd4ml-mcaip-kflw-eu-central-1-prod-mcaip-kflw/test/model/model.joblib

Apply the InferenceService configuration:

Deploy the InferenceService

kubectl apply -f inferenceservice.yml

Step 2: Create the Ingress Resource with TLS

Create an ingress.yml file to expose your InferenceService through Traefik ingress.

The following DNS formats are supported for Traefik ingress exposure.

Create ingress.yml

⚠️ Important

The ingressClassName should be traefik-private for intranet access.

Add the cert-manager.io/cluster-issuer: traefik-private-bmwca annotation to generate the TLS certificate automatically.

DNS Format

Private Subdomain

RoW: *.<env>.<product>.hub.orbit.<region>.aws.cloud.bmw

China: *.<env>.<product>.hub.orbit.cn-north-1.aws.unicom.cloud.bmw

Public Subdomain

RoW: *.<env>.<product>.hub.orbit.<region>.connected.bmw

China: *.<env>.<product>.hub.orbit.<region>.cv.bmw.com.cn

Ingress host value can get from isvc URL.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sklearn-iris-intranet
  namespace: mcaip-kflw
  annotations:
    cert-manager.io/cluster-issuer: traefik-private-bmwca
spec:
  ingressClassName: traefik-private
  rules:
    - host: sklearn-iris-intranet.mcaip-kflw.prod.mcaip-kflw.hub.orbit.eu-central-1.aws.cloud.bmw
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: istio-ingress
                port:
                  name: http
  tls:
    - hosts:
        - sklearn-iris-intranet.mcaip-kflw.prod.mcaip-kflw.hub.orbit.eu-central-1.aws.cloud.bmw
      secretName: sklearn-iris-intranet-tls

Apply the ingress configuration:

Deploy the ingress

kubectl apply -f ingress.yml

Step 3: Test the Endpoint

Once the ingress is deployed and the TLS certificate has been issued, test your inference endpoint.

Testing from Within the Intranet

Test inference endpoint from intranet

Send a test request directly to your endpoint:

curl -v -i -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [
      {
        "name": "input-0",
        "shape": [2,4],
        "datatype": "FP32",
        "data": [
          [6.8,2.8,4.8,1.4],
          [6.0,3.4,4.5,1.6]
        ]
      }
    ]
  }' \
  https://sklearn-iris-intranet.mcaip-kflw.prod.mcaip-kflw.hub.orbit.eu-central-1.aws.cloud.bmw/v2/models/sklearn-iris-intranet/infer

Replace the hostname with your actual inference service endpoint.

Deployment to the Public Internet

Exposing your inference endpoint to the public internet requires additional security considerations.

Authentication is mandatory for public endpoints.

Refer to the Kong Gateway Guidelines for more details:

https://orbit.bmwgroup.net/docs/api-management/kong-gateway#guidelines

Step 1: Deploy the Public InferenceService

Deploy InferenceService with internet label

⚠️ Important

For public internet access, your InferenceService must include the inferenceType: traefik-internet label.

This label instructs Knative to generate a VirtualService using the public Traefik domain configuration.

Create inferenceservice-internet.yml:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris-internet
  namespace: mcaip-kflw
  labels:
    inferenceType: traefik-internet
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  predictor:
    serviceAccountName: default-editor
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://cd4ml-mcaip-kflw-eu-central-1-prod-mcaip-kflw/test/model/model.joblib

Apply the configuration:

kubectl apply -f inferenceservice-internet.yml

Step 2: Generate Authentication Credentials

Create authentication credentials by generating a password hash for your users.

On Linux/macOS

Generate basic auth credentials on Unix-like systems

Use htpasswd to generate a password file:

htpasswd -c sklearn-iris-internet-auth foo

New password: <YOUR-FOO-PASSWORD>
Re-type new password: <YOUR-FOO-PASSWORD>

This creates an sklearn-iris-internet-auth file containing the encrypted credentials for user foo.

On Windows

Generate basic auth credentials on Windows

If htpasswd is not available, download Apache HTTPD binaries and extract them.

Then run:

htpasswd.exe -c auth foo

Follow the prompts to enter your password twice.

Step 3: Create the Kubernetes Secret

Create Kubernetes secret for authentication

Create a Kubernetes secret from the auth file:

kubectl create secret generic sklearn-iris-internet-basic-auth \
  --from-file=sklearn-iris-internet-auth \
  -n mcaip-kflw

Verify the secret was created correctly:

kubectl get secret sklearn-iris-internet-basic-auth -o yaml -n mcaip-kflw

Expected output:

apiVersion: v1
data:
  sklearn-iris-internet-auth: Zm9vOiRhcHIxJExERVhxR1RNJE1ncmRJUHRZZ3JkT2Q5cVFxUTZweTEK
kind: Secret
metadata:
  name: sklearn-iris-internet-basic-auth
  namespace: mcaip-kflw
type: Opaque

Step 4: Create the Traefik Middleware

Traefik uses Middleware CRDs for authentication instead of ingress annotations.

Create middleware.yml

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: sklearn-iris-internet-basic-auth-middleware
  namespace: mcaip-kflw
spec:
  basicAuth:
    secret: sklearn-iris-internet-basic-auth

Apply the middleware:

kubectl apply -f middleware.yml

Step 5: Create the Public Ingress with Authentication

Create an ingress resource that references the Traefik Middleware.

Create ingress.yml with authentication

⚠️ Important

For public access, add:

cert-manager.io/cluster-issuer: traefik-public-letsencrypt

The middleware annotation format is:

{namespace}-{middleware-name}@kubernetescrd

DNS formats are the same as described earlier.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sklearn-iris-internet-basic-auth
  namespace: mcaip-kflw
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: mcaip-kflw-sklearn-iris-internet-basic-auth-middleware@kubernetescrd
    cert-manager.io/cluster-issuer: traefik-public-letsencrypt
spec:
  ingressClassName: traefik-public
  rules:
    - host: sklearn-iris-internet.mcaip-kflw.prod.mcaip-kflw.hub.orbit.eu-central-1.connected.bmw
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: istio-ingress
                port:
                  name: http
  tls:
    - hosts:
        - sklearn-iris-internet.mcaip-kflw.prod.mcaip-kflw.hub.orbit.eu-central-1.connected.bmw
      secretName: sklearn-iris-internet-basic-auth-tls

Step 6: Deploy the Ingress

Apply authenticated ingress to cluster

kubectl apply -f ingress.yml

Step 7: Test the Authenticated Endpoint

Test endpoint with basic authentication

Send a test request with your credentials:

curl -v -i -X POST \
  -H "Content-Type: application/json" \
  -u 'foo:<YOUR-FOO-PASSWORD>' \
  -d '{
    "inputs": [
      {
        "name": "input-0",
        "shape": [2,4],
        "datatype": "FP32",
        "data": [
          [6.8,2.8,4.8,1.4],
          [6.0,3.4,4.5,1.6]
        ]
      }
    ]
  }' \
  https://sklearn-iris-internet.mcaip-kflw.prod.mcaip-kflw.hub.orbit.eu-central-1.connected.bmw/v2/models/sklearn-iris-internet/infer

Replace <YOUR-FOO-PASSWORD> with the password you created earlier and update the hostname to match your configuration.

Introduction​

Deploying Your Model via KServe​

Understanding KServe Deployment​

Step 1: Create the InferenceService Configuration​

Step 2: Integrate the InferenceService into Your Pipeline​

Exposing the Inference Endpoint​

Prerequisites​

ORBIT-USE CLI Setup​

Install the orbit-use CLI​

Generate Default Credentials​

Deploying to the Intranet​

Step 1: Deploy the InferenceService​

Step 2: Create the Ingress Resource with TLS​

Step 3: Test the Endpoint​

Testing from Within the Intranet​

Deployment to the Public Internet​

Step 1: Deploy the Public InferenceService​

Step 2: Generate Authentication Credentials​

On Linux/macOS​

On Windows​

Step 3: Create the Kubernetes Secret​

Step 4: Create the Traefik Middleware​

Step 5: Create the Public Ingress with Authentication​

Step 6: Deploy the Ingress​

Step 7: Test the Authenticated Endpoint​