Skip to main content

Architecture

note

The Inference API is designed as a scalable platform for orchestrating ML model deployments and serving inference requests on EKS. For this architecture, we use Ray Serve for the creation of Live Inference Applications to serve ML models in real time, and Ray Data for Batch job processing, alongside cloud services for model and metadata storage.

The core components of the architecture include:

  1. Inference API: A FastAPI-based RESTful API that serves as the primary interface for users to submit and manage inference jobs. It handles authentication and request validation, orchestrating Ray Serve deployments for Live Inference and submitting Ray Jobs for Batch Inference.

  2. MLFlow: Used to store ML models as artifacts, exposing model URIs that Ray uses to retrieve models from S3 for serving and batch processing.

  3. Ray Clusters: Responsible for executing inference workloads — Ray Serve hosts Live Inference application deployments, while Ray Data processes Batch Inference jobs.

  4. AWS Storage Layer: Storage solution for Live and Batch Inference job metadata (DynamoDB) and model artifacts (S3).

Architecture Diagram​

Inference API Architecture