Skip to main content

Setting Up Git Access and Poetry for Kubeflow Development

Introduction​

Before you can clone repositories and begin working with Kubeflow pipelines, you need to set up two essential components: Git authentication using a Personal Access Token, and Poetry for managing your Python project dependencies. This guide walks you through generating a secure Git token, understanding Poetry's role in your workflow, and verifying that your environment is properly configured. These setup steps are critical for ensuring secure access to your repositories and maintaining reproducible, well-managed Python environments.

Understanding Git Authentication​

Why Git Access Tokens Are Necessary​

To clone repositories from bmw.ghe.com (GitHub Enterprise), you need a secure authentication method. Password-based authentication has been deprecated in Git, so you must use a Personal Access Token (PAT) instead. This token acts as a secure credential that authenticates your requests when cloning or pushing code to repositories.

Generating a Personal Access Token​

click here for the steps to generate a PAT

Step 1: Access Developer Settings​

Navigate to your account settings on bmw.ghe.com and locate the developer settings section.

Step 2: Create a New Token​

In the developer settings, click the "Generate New Token" button to create a new Personal Access Token.

Step 3: Configure Token Permissions​

When creating your token, ensure you grant it the following permissions:

  • repo: Full control of private repositories

For a more detailed explanation of token permissions, refer to GitHub's official guide.

Step 4: Save Your Token Securely​

After generating the token, GitHub will display it once. Copy this token immediately and store it securely.

Important Security Reminders​

  • Never share your token with anyone. Your Personal Access Token is tied directly to your personal user account, and anyone with access to it could make changes to your repositories.
  • Store your token in a password manager for safekeeping. Since Kubeflow Notebooks can be accessed by multiple users and there is currently no secure way to store credentials within them, you'll need to retrieve your token each time you push changes.
  • Treat your token like a password — if you accidentally expose it, regenerate it immediately from your GitHub settings.

Understanding Poetry​

What is Poetry?​

Poetry is a modern dependency management and packaging tool for Python projects. It simplifies the entire process of managing project dependencies, creating reproducible environments, and packaging your code for distribution. Instead of manually juggling multiple configuration files, Poetry centralizes everything into a single pyproject.toml file.

Poetry vs. Traditional Package Managers​

Poetry significantly improves upon traditional tools like pip and requirements.txt:

FeaturePoetrypip
Configuration FileUses pyproject.toml for all project configuration and dependenciesUses requirements.txt for dependencies and setup.py for metadata
Dependency ResolutionAutomatically resolves all dependencies and sub-dependencies, creating a poetry.lock file for reproducibilityInstalls listed dependencies without resolving sub-dependencies
Virtual EnvironmentsAutomatically creates and manages virtual environments for each projectRequires manual creation and management using venv or virtualenv
ReproducibilityEnsures reproducible builds across different systems with the poetry.lock fileNo built-in mechanism for ensuring reproducible builds
PublishingSimplifies packaging and publishing projects to PyPIRequires additional tools like twine for publishing
Dependency SpecificationSupports flexible version constraints (e.g., ^1.0.0, ~1.0)Typically uses exact version numbers or simple ranges

Key Benefits for Your Workflow​

  • Reproducible Environments: Everyone working on your project will have identical dependencies
  • Simplified Setup: One command to install all dependencies instead of multiple steps
  • Automatic Dependency Resolution: Avoids conflicts between package versions
  • Lock File: The poetry.lock file ensures consistent installations across all environments

Installing and Verifying Poetry​

Step 1: Check if Poetry is Installed​

Before installing Poetry, check if it's already available on your system. Run the following command in your terminal:

poetry

If Poetry is installed correctly, this command will display information about Poetry along with helpful commands and usage examples.

Step 2: Troubleshooting Installation Issues​

If you see an error message stating that Poetry is not found, you need to install it.

Step 3: Install Poetry​

If Poetry is not installed, install it using the official installation guide:

Visit: https://python-poetry.org/docs/#installation

Follow the installation instructions for your operating system (Windows, macOS, or Linux).

Step 4: Verify Installation​

After installation, run the verification command again to confirm Poetry is working:

poetry

You should see the Poetry help information and available commands.

Cloning Your First Repository and Setting Up Your Project​

Cloning the Starter Template Repository​

What is the Template Repository?

The cd4ml-pipelines-template repository contains the starter code and structure for creating Kubeflow pipelines. This template provides a pre-configured project structure, example components, and configuration files that you'll use as the foundation for your own pipeline project.

Method 1: Cloning via Terminal​

If you prefer using the command line, you can clone the repository directly using Git.

Step 1: Clone the Repository​

Open a terminal in your notebook environment and run the following command:

git clone https://<PAT>@bmw.ghe.com/connected-ai/cd4ml-pipelines-template

Replace <PAT> with the Personal Access Token you generated earlier.

Example:

git clone https://ghp_1234567890abcdefghijklmnopqrstuvwxyz@bmw.ghe.com/connected-ai/cd4ml-pipelines-template

This command will download the entire repository to your current directory.

Method 2: Cloning via the VS Code Interface​

If you prefer a graphical approach, VS Code provides a built-in Git interface.

Step 1: Access the Git Panel​

In your VS Code notebook environment, locate the Git icon in the left sidebar menu (it looks like a branching diagram).

Step 2: Clone the Repository​

Click on the Git icon and look for the "Clone Repository" button. This will open a dialog box where you can enter the repository URL.

Step 3: Enter the Clone URL​

In the dialog box, enter the HTTPS clone address:

https://bmw.ghe.com/connected-ai/cd4ml-pipelines-template

You can find this URL on the GitHub repository page by clicking the Code button.

Step 4: Provide Credentials​

When prompted for credentials:

  • Username: Enter any identifier (e.g., your Q-number or email)
  • Password: Enter your Personal Access Token (the one you generated earlier)
Step 5: Verify Successful Clone​

Once the clone completes successfully, you'll see all the repository files displayed in the left sidebar file explorer.

Setting Up the Template Environment​

Step 1: Navigate to the Cloned Repository​

Change to the cloned repository directory:

cd cd4ml-pipelines-template

After this command, your current path should be:

/home/jovyan/cd4ml-pipelines-template

Step 2: Configure Poetry for Your Environment​

Before installing dependencies, configure Poetry to use your Personal Access Token and to store virtual environments locally within the project.

Configure CAIP SDK Authentication​

Poetry needs to authenticate with the private CAIP SDK package repository. Run the following command, replacing <YOUR-PAT> with your Personal Access Token:

poetry config http-basic.caip-sdk build <YOUR-PAT>
Configure Virtual Environment Location​

Configure Poetry to create the virtual environment inside your project directory (rather than a centralized location):

poetry config virtualenvs.in-project true

Step 3: Install Dependencies​

Install all project dependencies using Poetry:

poetry install
What happens during poetry install?

This command reads the pyproject.toml file, resolves all dependencies, and creates a poetry.lock file to ensure reproducibility. It will also automatically create a virtual environment named .venv in your project directory.

Step 4: Activate the Virtual Environment​

Activate the Poetry-managed virtual environment based on your operating system:

For Linux, macOS, and Notebooks:

source .venv/bin/activate

For Windows:

.venv\Scripts\activate

After activation, your terminal prompt will display (.venv) at the beginning, confirming the virtual environment is active.

Step 5: (Optional) Install Development and Testing Dependencies​

If you plan to develop on this template or run tests, install the optional development packages:

poetry install --with dev,test

This installs additional packages useful for development, testing, and code quality checks.

Creating Your Custom Project from the Template​

Now that you have the template repository set up, you'll use Copier to generate your own custom project based on this template.

Step 1: Return to the Parent Directory​

Navigate to the parent directory where you want to create your new project:

cd ..

You should now be in the /home/jovyan directory.

Step 2: Create a Project Directory​

Create a new directory for your custom project. Replace <my_new_project_name> with your desired project name:

mkdir <my_new_project_name>

Example:

mkdir my_ml_pipeline

Step 3: Enter Your Project Directory​

Change into your newly created project directory:

cd <my_new_project_name>

After this command, your path should be:

/home/jovyan/<my_new_project_name>

Step 4: Generate Your Project with Copier​

Copier is a tool that generates new projects from templates. Use it to create your custom project from the template repository. Choose either the HTTPS or SSH method:

Using HTTPS with Personal Access Token:

copier copy https://bmw.ghe.com/connected-ai/cd4ml-pipelines-template.git .

Using SSH (if you have SSH keys configured):

copier copy git@bmw.ghe.com:connected-ai/cd4ml-pipelines-template.git .
How do I specify a template version?

If you want to use a specific version of the template, add the --vcs-ref parameter:

copier copy https://bmw.ghe.com/connected-ai/cd4ml-pipelines-template.git . --vcs-ref=<my_version>

Step 5: Configure Your Project Parameters​

Copier will prompt you to enter project-specific information. Provide the following inputs:

Product Name:

ai lab

(Replace with your product or project name)

Namespace:

astroboy

(Replace with your Kubernetes namespace)

Non-Production AWS Account ID:

974677654443

(This is the AI Lab dev account. Replace with your designated AWS account ID if different)

Test AWS Region:

eu-central-1

(Replace with your AWS region if different)

These parameters customize the generated project configuration for your specific environment.

Setting Up Your Custom Project Environment​

Step 1: Enter Your Project Directory​

Navigate to your newly created custom project directory:

cd <my_new_project_name>

Step 2: Deactivate the Template Virtual Environment​

If you still have the template's virtual environment active, deactivate it first:

deactivate

Step 3: Install Project Dependencies​

Install the dependencies for your new custom project using Poetry:

poetry install

This command reads your project's pyproject.toml file and installs all required packages and dependencies specific to your project.

Step 4: Activate Your Project's Virtual Environment​

Activate the virtual environment for your custom project:

For Linux, macOS, and Notebooks:

source .venv/bin/activate

For Windows:

.venv\Scripts\activate

Step 5: (Optional) Install Development and Testing Packages​

If you want to include development and testing tools for your project:

poetry install --with dev,test

Understanding Git Fundamentals​

What is Git?

Git is a distributed version control system that tracks changes to your code throughout the development process. It enables multiple developers to work on the same project simultaneously without interfering with each other's work. Git records every modification made to your codebase, allowing you to review history, revert to previous versions, collaborate seamlessly, and manage different development branches.

Core Git Concepts​

Repository

A repository (repo) is a directory containing your project's files along with the complete revision history of those files. It stores all commits, branches, and metadata about your project. When you clone a repository, you download the entire project with all its history.

Commit

A commit is a snapshot of your repository at a specific point in time. Each commit has a unique identifier (hash) and includes a descriptive message explaining what changes were made. Commits form a timeline of your project's evolution and allow you to revert to previous states if needed.

Branch

A branch is a separate line of development that diverges from the main codebase. Branches allow you to work on new features or bug fixes without affecting the main code. The default branch is typically called main or master. You can create multiple branches to work on different tasks simultaneously.

Merge

Merging combines changes from one branch into another branch. This is typically used to integrate completed features from development branches back into the main branch. Git automatically attempts to merge changes, but conflicts may occur if different branches modified the same code.

Remote Repository

A remote repository is a version of your project hosted on the internet or a network server. Popular platforms include GitHub (including GitHub Enterprise like bmw.ghe.com), GitLab, and Bitbucket. The primary remote is typically called origin and represents the central repository where your team stores code.

Essential Git Commands​

View all Git commands

Understanding these fundamental Git commands will help you manage your code effectively:

  • git init: Initializes a new Git repository in your current directory
  • git clone: Creates a local copy of a remote repository on your machine
  • git status: Shows the current state of your working directory and staging area
  • git add: Stages changes to be included in the next commit
  • git commit: Records staged changes to your local repository with a descriptive message
  • git push: Uploads your local commits to the remote repository
  • git pull: Fetches and integrates changes from the remote repository into your local branch

The Clone Operation Explained​

What happens when you clone a repository?

What Cloning Does

Cloning is the process of downloading a complete copy of a remote repository to your local machine. When you clone, you receive:

  • All project files in their current state
  • The entire revision history and all previous commits
  • All branches from the remote repository
  • Git configuration that automatically tracks the original repository

What Happens During Clone

When you execute a clone command, Git performs several actions automatically:

  1. Downloads Repository Content: All project files are transferred to your local machine
  2. Sets Up Remote Tracking: Git creates a reference called origin pointing to the original repository URL
  3. Checks Out Default Branch: The main branch (usually main or master) is checked out and ready for use
  4. Creates Git Metadata: A .git directory is created containing all version history and configuration

Starting Work After Cloning

Once a repository is cloned, you can immediately begin working on it:

  • The default branch is already checked out
  • You have the complete history available locally
  • You can create new branches for your work
  • You can make commits and eventually push changes back to the remote repository

Your environment is now ready. You can begin developing your Kubeflow pipeline components and building your machine learning workflow. Proceed to the next section of the guide to start creating your pipeline.