Multi-Profile Configuration

Create one base compose file, add a production profile, inspect the merged configuration, and deploy it safely.

Goal

By the end of this tutorial, you will have a default configuration for development and a prod profile that overrides only the production differences.

What You Will Create

  • mlknife-compose.yaml - the default profile, used for local or development settings.
  • mlknife-compose.prod.yaml - the production profile, selected with --profile prod.

What Is a Profile?

A ModelKnife config profile is a named compose file variant. The default profile lives in mlknife-compose.yaml. A named profile lives in mlknife-compose.<profile>.yaml.

What

A profile changes the compose configuration that ModelKnife loads.

Why

Profiles avoid copying a full compose file just to change a few fields.

How

Select a profile with --profile prod. Use -p prod as shorthand after you are comfortable with the command.

Profile Is Not Workspace

--profile selects configuration. --workspace selects deployment isolation. For example, --profile prod loads production settings, while --workspace prod deploys into the production workspace. Learn the full model in Workspaces.

File Naming

Use dot-style filenames for profiles. The part between mlknife-compose. and .yaml is the profile name.

Profile file names 
mlknife-compose.yaml                    # default profile
mlknife-compose.prod.yaml               # profile: prod
mlknife-compose.batch-training.yaml     # profile: batch-training
mlknife-compose.daily-scoring.yaml      # profile: daily-scoring

Step 1: Create the Default Config

Start with the smallest useful base file. This file is loaded when no profile is selected.

1

Minimal Default Profile

Use development-safe values in mlknife-compose.yaml.

mlknife-compose.yaml 
name: image-classifier
backend: aws

parameters:
  environment: dev
  data_path: s3://example-${parameters.environment}/images

services: {}
modules: {}

Step 2: Add a Production Profile

Create mlknife-compose.prod.yaml. Keep only the fields that are different in production.

2

Minimal Production Override

The profile inherits the default config, then overrides matching fields.

mlknife-compose.prod.yaml 
parent_profile: default

parameters:
  environment: prod

Why Include parent_profile?

parent_profile: default is optional because default is assumed, but keeping it in the first examples makes the inheritance relationship obvious.

Step 3: Understand the Result

When you load --profile prod, ModelKnife first loads mlknife-compose.yaml, then loads mlknife-compose.prod.yaml on top of it. The profile inherits anything it does not mention, and overrides matching fields it does mention.

3

Merged Configuration

This is the effective configuration ModelKnife sees for --profile prod.

Effective prod configuration 
name: image-classifier
backend: aws

parameters:
  environment: prod
  data_path: s3://example-prod/images

services: {}
modules: {}

Step 4: Inspect and Deploy

Use the long flags while learning. -p is shorthand for --profile, and -w is shorthand for --workspace.

Discover and inspect profiles 
mk list-profiles
mk s show --profile prod
mk p status --profile prod
Deploy with the prod profile 
mk s deploy --profile prod
mk p deploy --profile prod

Production Isolation

A production profile changes configuration. A production workspace changes where state and cloud resources are isolated. For real production deployments, use both intentionally.

Deploy prod config into prod workspace 
mk s deploy --workspace prod --profile prod
mk p deploy --workspace prod --profile prod
mk p status --workspace prod --profile prod

Common Variation: Pipeline Profiles

A profile does not have to represent an environment. It can also represent a pipeline variant that shares base services and defaults.

Pipeline profile examples 
mlknife-compose.batch-training.yaml
mlknife-compose.daily-scoring.yaml

mk p deploy --profile batch-training
mk p run --profile daily-scoring

Advanced Parent Profiles

A profile can inherit from another profile by setting parent_profile to that profile name. Use this sparingly; deep inheritance chains are harder to reason about.

Realistic Example

Once the basic idea is clear, a production profile usually changes resource names, storage settings, workers, or module parameters.

mlknife-compose.yaml 
name: image-classifier
backend: aws

parameters:
  environment: dev
  data_path: s3://example-${parameters.environment}/images

services:
  artifacts_bucket:
    type: object_storage_bucket
    configuration:
      bucket_name: image-classifier-${parameters.environment}-artifacts

modules:
  prepare_training_data:
    type: etl
    repository: ../src
    entry_point: jobs/prepare_training_data.py
    job_parameters:
      input_path: ${parameters.data_path}/raw
      output_path: ${parameters.data_path}/prepared
mlknife-compose.prod.yaml 
parent_profile: default

parameters:
  environment: prod

services:
  artifacts_bucket:
    configuration:
      versioning: true
      lifecycle:
        expire_after_days: 365

Troubleshooting

Profile Not Found

Check the filename. mlknife-compose.prod.yaml maps to --profile prod.

Unexpected Value Still Appears

The profile inherits from mlknife-compose.yaml. Override that field explicitly. If a section should be empty, clear it with modules: {} or services: {}.

Deployment Went to the Wrong Place

Remember that --profile changes configuration, while --workspace changes deployment isolation.

Best Practices

  • Use one naming convention: document and create profiles as mlknife-compose.<profile>.yaml.
  • Keep profiles small: a profile should show what is different from the base file.
  • Name by intent: use names such as prod, batch-training, or daily-scoring.
  • Use workspaces for isolation: use --workspace prod for production and personal workspaces for developer testing.
  • Keep secrets out of YAML: use provider-native secret stores or environment-based credential mechanisms.
  • Review before production: inspect the plan or deploy output before applying a production workspace change.
Production Deployment
Advanced ML Pipelines