Configuration Guide

ModelKnife has two main configuration areas: AWS Profile Setup for connecting to your AWS account, and Project Configuration for defining your ML pipelines and services.

Quick Start

New to ModelKnife? Start here:

  1. Create your first project: Create mlknife-compose.yaml
  2. Initialize team setup: mk setup init
  3. Set up AWS access: mk conf set your-aws-profile (if needed)

1. Project Configuration

Start by creating your ML project configuration. ModelKnife uses YAML files to define pipelines and services.

Basic Project Structure

mlknife-compose.yaml 
# mlknife-compose.yaml
name: "my-ml-pipeline"
author: "ml-team"
version: "v1.0"

parameters:
  environment: dev
  data_bucket: "ml-data-${parameters.environment}"

executors:
  # Compute environment templates

services:
  # Infrastructure services

modules:
  # ML processing modules

Common Project Commands

# Validate configuration format
mk fmt --check

# Deploy services and infrastructure
mk service deploy

# Deploy and run ML pipelines
mk pipeline deploy
mk pipeline run my-pipeline

# Check deployment status
mk service status
mk pipeline status

2. AWS Profile Configuration (Advanced)

For advanced AWS profile management or multiple environments, use ModelKnife's profile configuration commands.

Profile Management Commands

# Set AWS profile (persists across sessions)
mk conf set production

# View current configuration
mk conf show

# Remove profile configuration (uses environment or default)
mk conf unset

Configuration Priority

ModelKnife checks for AWS profile configuration in this order:

  1. Config file - ~/.mlknife/config.json (set via mk conf set)
  2. Environment variable - MLKNIFE_AWS_PROFILE
  3. Default - AWS default profile or environment credentials

Multi-Environment Workflow

# Development workflow
mk conf set development
mk service deploy
mk pipeline run data-processing

# Switch to production
mk conf set production
mk service deploy

# Temporary override with environment variable
MLKNIFE_AWS_PROFILE=testing mk service deploy

# Check current configuration
mk conf show

3. Troubleshooting

AWS Profile Issues

Common Issues and Solutions

  • Profile not found: Verify the profile exists in ~/.aws/credentials or ~/.aws/config
  • Permission denied: Check that the profile has the necessary AWS permissions for ModelKnife operations
  • Wrong account: Use aws sts get-caller-identity --profile <profile-name> to verify the profile points to the correct AWS account

Debugging Commands

# Check available AWS profiles
aws configure list-profiles

# Check ModelKnife's current configuration
mk conf show

# Test profile credentials manually
aws sts get-caller-identity --profile production

# Verify ModelKnife can access AWS
mk conf set production
mk conf show  # Should display AWS account information

Next Steps

Now that you understand configuration basics, explore these topics:

Learn More