Skip to main content

Migrating from Dedicated Clusters to Managed Kubeflow

This guide explains how to migrate your machine learning projects from Dedicated Kubeflow Clusters to Managed Kubeflow. Previously, use cases often managed multiple dedicated clusters for different stages (e.g., TEST, INT, E2E, PROD). Managed Kubeflow now provides a unified, production-grade environment with a modern staging concept, eliminating the need for separate infrastructure instances for each stage. Migrating ensures your projects are future-proof, easier to manage, and benefit from platform support and automation.

The migration steps are the same as for AI-Research, but with one key difference: you must map your previous staging approach—where each environment (TEST, INT, E2E, PROD) was a separate cluster—to the unified staging concept in Managed Kubeflow. Now, all stages are managed within a single infrastructure using vertical staging for use cases. Please read the staging concept documentation to understand how to organize your pipelines and models using the new approach.

Migration Steps​

1 - Complete Onboarding Prerequisites​

Before you begin the migration, make sure you have completed all onboarding prerequisites. This includes reviewing the onboarding requirements, creating and linking your own APP-ID, setting up your own GitHub organization on ATC GitHub to manage your code repositories, and creating a CDH environment to connect your use case or data provider. These steps ensure you have the necessary access and resources to work in Managed Kubeflow.

2 - Understand the New Staging Concept​

Managed Kubeflow introduces a new staging concept for organizing pipelines and models within a single, production-grade infrastructure. This new approach replaces the old environment-based model and makes it easier to move from experimentation to production. To get familiar with this, you should read about the staging concept, which explains how to stage Kubeflow pipelines and models using MLflow.

3 - Set Up Your Managed Kubeflow Workspace​

Next, you need to request a new CAIP space and a Managed Kubeflow workspace, which will be the new home for your migrated workloads. Follow the onboarding to Managed Kubeflow guide to request your workspace. Once your workspace is ready, you should link your GitHub repositories using CI Trust, as described in the guide to link your GitHub repo. You will also need to link your CDH use case or provider using the CDH Connection. If you have not yet migrated your data connections, follow the setup CDH connection guide and the guide to link your CDH environment.

4 - Migrate Your Use Case Code (Notebooks, Pipelines, Endpoints, Infrastructure)​

With your workspace set up, you can now systematically migrate your project assets and workflows from your dedicated clusters to Managed Kubeflow. This comprehensive migration involves moving code, data connections, models, and infrastructure components while adapting from your multi-cluster setup (TEST, INT, E2E, PROD) to the unified staging approach. Use the following checklist to ensure a complete and successful migration:

Code and Development Environment​

Jupyter Notebooks:

  • Export your Jupyter notebooks from your dedicated clusters and import them into your new Managed Kubeflow workspace
    • After logging into the Web-UI, navigate to the Notebooks page and connect to the notebooks to be exported. In the JupyterLab interface, right-click on the notebook file, then select Download to save it locally
    • In Managed Kubeflow, open your new JupyterLab workspace and use the Upload Files button (upload icon in the file browser) to upload your downloaded notebooks- Update any hardcoded paths or configurations to match the new environment
  • Test notebook execution to ensure compatibility with the new infrastructure

VS Code Server:

  • Commit any untracked code in your current VS Code environment to avoid losing work
  • Create a new VS Code server instance in the Managed Kubeflow cluster
  • Clone your repositories to the new server instance and leverage the same development experience as before
  • Check if your use cases have data or files stored in volumes attached to the old pods that need migration - these may require manual export and import

Kubeflow Pipelines and Workflows​

Pipeline Migration:

  • Update your caip-sdk configuration with the new AWS account and role assignments for your Managed Kubeflow environment. The central configuration is available under config/config.yaml as described here.
  • Adapt your pipelines to use the new vertical staging approach instead of separate clusters for each environment
  • Deploy your pipelines and test their executions to verify they work correctly in the new setup
  • For automatic deployment of the pipelines, update role assumption configurations in your GitHub CI/CD pipelines and ensure CI Trust is properly configured
  • Migrate any recurring pipeline runs and schedules from all your previous clusters (TEST, INT, E2E, PROD) to the new unified environment

Data Connections:

  • Update your data connections to use the new CDH setup as configured in step 3
  • Consolidate S3 bucket connections from your multiple clusters and update access permissions as needed
  • Ensure that stage-specific data access patterns are maintained using the new staging concept

Data and Applications​

Streamlit Applications:

  • Redeploy your Streamlit applications using the CI/CD pipeline in the new environment
  • Update application configurations and data connections as needed
  • Migrate applications from all previous environments (TEST, INT, E2E, PROD) and adapt them to use the new staging approach
  • Test application functionality and user access

Data:

  • If you have data in S3 across multiple clusters or you are a GenAI use case using pgvector, contact us to support you with migrating the existing data
  • Consolidate data from your previous clusters where appropriate
  • Test data access and connectivity from your new workspace

Models and Serving​

MLflow Models:

  • Migrate your MLflow models and experiments from all your dedicated clusters to the new environment
  • Consolidate models from different stages (TEST, INT, E2E, PROD) using MLflow's staging capabilities

KServe Model Serving:

  • Redeploy your model serving endpoints from all dedicated clusters on the new Managed Kubeflow cluster
  • Update endpoint configurations and ensure proper scaling settings
  • Adapt your promotion workflow from cluster-based to stage-based using the new staging concept
  • Test model inference endpoints to verify they respond correctly
  • Update any client applications or services that consume these endpoints

Katib (Hyperparameter Tuning):

  • If you are using Katib for hyperparameter optimization in your dedicated clusters, note that this is currently a proof-of-concept feature in Managed Kubeflow
  • Katib is only deployed on a request basis - contact us to activate it if you need this functionality
  • Migrate your Katib experiments and configurations once the service is activated

Infrastructure and Configuration​

Container Images:

  • ECR Repositories:
  • Request new ECR repositories through Orbit self-service for your container images
  • Migrate your custom images from your dedicated cluster ECR repositories to the new ECR repositories
  • Update your CI/CD pipelines to push to the new ECR repositories
  • Ensure all image references in your deployments point to the new registry

Security and Access:

  • Migrate secrets, configmaps, and other Kubernetes resources from all your dedicated clusters to your new Managed Kubeflow namespace
  • Review your existing custom policies across different roles from each cluster and consolidate them into the new cluster using the correct AWS account IDs and roles through PRs in the products repo
  • Adapt environment-specific policies to work with the new vertical staging approach
  • If you are currently using Amazon Bedrock in any of your clusters, migrate to using our LLM API service

Testing and Validation​

Once all components are migrated:

  • Run end-to-end tests of your complete workflows to ensure everything functions correctly in the unified environment
  • Validate that the new staging approach provides the same isolation and promotion capabilities as your previous multi-cluster setup
  • Test data flow from ingestion through model training to serving across all stages
  • Verify monitoring and observability features are capturing metrics as expected
  • Confirm that all team members have appropriate access to the new environment
Migration Checklist

Use this section as a comprehensive checklist during your migration. Check off each item as you complete it to ensure nothing is missed during the transition from Dedicated Clusters to Managed Kubeflow. Pay special attention to consolidating resources from multiple clusters (TEST, INT, E2E, PROD) into the single unified environment with vertical staging.

Need Support?​

If you need help with any migration tasks or encounter issues, you can always reach out to our support team via ITSM Next. Simply open an incident ticket for the "Connected AI Platform" service offering. For detailed, step-by-step instructions on how to raise a support request, please see our Support page. Our team is available to assist you at any stage of your migration to Managed Kubeflow.