Skip to main content

CDH Connection

Context and IAM Setup​

As the Connected AI Platform (CAIP) transitions to a managed architecture, the integration with the Cloud Data Hub (CDH) will adopt a more scalable setup. This new approach will utilize role chaining and CDH-managed environments. It is essential to outline the new IAM architecture, which describes how CAIP applications interact with CDH resources using roles in both CAIP and CDH accounts. This includes access to Personally Identifiable Information (PII) APIs through secrets and specific roles (DEPERSO / REPERSO). Additionally, we will refine the IAM policies on CDH accounts to enhance security and prevent unauthorized access to AWS resources not included in the CDH offering. This document details the new architecture and the necessary IAM policy declarations for both CAIP and CDH roles.

The diagram below illustrates how CAIP applications access CDH dataset resources through cross-account role chaining from CAIP accounts to CDH environment accounts.

CDH IAM Architecture

Key Points of the Architecture​

  • Example Accounts: The AWS account named MCAIP-TRAIN serves as an example for a CAIP product, while MCAIP-TRAIN-INT-MANAGED is designated as a CDH-managed environment account for the MCAIP-TRAIN product.

  • Consistent Role Naming: CAIP will utilize a single role on CDH environment accounts called caip-role-managed-cdh. This role name remains constant, regardless of the product name or environment.

  • Role Permissions: On CAIP accounts, the roles assigned to Kubeflow, Jupyter, and Kubernetes applications (denoted as *-cd4ml) must include a policy that allows the action sts:AssumeRole for the caip-role-managed-cdh on the CDH environment account (e.g., MCAIP-TRAIN-INT-MANAGED).

    CDH Environment

  • Execution Roles for Managed Services: For CAIP workloads using AWS managed services like Glue or SageMaker, their execution roles must also have a policy allowing sts:AssumeRole for the caip-role-managed-cdh on CDH.

  • CDH Resource Access: The caip-role-managed-cdh on CDH accounts must have an IAM policy that permits access to S3, Glue, KMS, and Athena resources. Detailed guidance on setting up these policies can be found in the CDH Documentation.

  • Accessing PII APIs: To use the PII API from CAIP, the workload must assume the caip-role-managed-cdh on CDH, which should have a policy allowing sts:AssumeRole for PII roles on account 607770795732 (specifically deperso-CDH_ENV_ACCOUNT_ID and reperso-CDH_ENV_ACCOUNT_ID). The CDH_ENV_ACCOUNT_ID corresponds to the CDH environment account ID (e.g., 539247460700 for MCAIP-TRAIN-INT-MANAGED). This role must also allow actions to retrieve the PII API Key stored in Secrets Manager.

Implications for Accessing CDH Resources​

Based on the policies outlined above, the following implications for accessing CDH resources can be noted:

  • The caip-role-managed-cdh can only be assumed by CD4M, developer and execution roles on CAIP accounts and by Glue sessions/jobs on CDH environments.

  • KMS must have Encrypt/Decrypt access to all keys because CDH environments can be used as dataset consumers from other environments/providers, which then have different KMS keys used to encrypt data.

  • The caip-role-managed-cdh can be used as an execution role for Glue sessions and jobs created on CDH environments.

  • All Glue actions are allowed by the policy. This means the role will be able to perform all Glue operations on CDH environments except deleting a dataset database, because Glue databases on CDH environments are only mirrors shared by the original CDH account where the dataset is created using LakeFormation.

  • All S3 operations are allowed. This means the role can read/write objects or perform any bucket operation if it's created on the CDH environment. If the CDH environment is a dataset provider, it can write data to the dataset bucket on the CDH account; if it's a dataset consumer in another environment/provider, it can only read data from the dataset bucket.

  • All Athena operations on the CDH environment are allowed.

  • For PII API operations, the caip-role-managed-cdh can assume DEPERSO and REPERSO roles on the PII API account and can read the PII API KEY secret value in Secrets Manager, which are all required to call the PII API.

Conclusion​

The described setup is expected to facilitate easier interaction between CAIP and CDH datasets through a scalable architecture that leverages CDH-managed environments and role chaining. This approach will reduce the operational burden of managing CDH resources on CAIP and prepare the IAM layer for the migration to a Managed CAIP architecture.

Further Reading​