ModelKnife has four primary architecture concepts: stack, pipeline, module, and service. Executors are supporting runtime templates for modules. Start with the stack: it is the deployable system boundary that contains the pipeline and services.

Key Distinction

Stack = one deployable ML system for an environment
Pipeline = workflow graph inside the stack
Module = one pipeline work unit
Service = long-running infrastructure used by applications or modules
Executor = reusable runtime template used by modules

Stack

Deployable ML System Boundary

What: The top-level ModelKnife system described by one compose file
When: Use one stack per product, workload, or environment boundary
Contains: Services, modules, executors, parameters, dependencies, and provider settings

Stack Configuration Shape 
name: recommendation-stack
author: ml-team
backend: aws

services:
  feature_store: { type: dynamodb_table }

executors:
  python_processor: { type: sagemaker_processor }

modules:
  build_features:
    executor: ${executors.python_processor}
Boundary: Defines what belongs to one deployable ML system
Environment-Aware: Profiles can customize dev, staging, and production
Provider-Backed: The stack can target AWS today and other providers over time
Lifecycle Split: Services and pipelines can be deployed independently

Pipeline

Workflow Graph inside a Stack

What: A deployable workflow assembled from modules and orchestrated as a dependency graph
When: Run scheduled or on-demand processing for training, feature generation, or batch inference
In YAML: The modules section defines the pipeline graph
Examples: Data ETL → feature engineering → model training → batch prediction

Pipeline Section Example 
name: recommendation-stack

modules:
  data_cleaning:
    executor: ${executors.glue_etl}
  feature_engineering:
    executor: ${executors.python_processor}
    depends_on: [data_cleaning]
  model_training:
    executor: ${executors.python_processor}
    depends_on: [feature_engineering]
Graph-Based: Module dependencies define execution order
Batch Processing: Handle large-scale data processing jobs
Orchestrated: The provider creates the runtime orchestration
Versioned: Pipeline deployments can be tracked separately from services

Module

Pipeline Work Unit

What: One named work unit in a pipeline graph
When: Each distinct task in your workflow, such as data cleaning, training, or inference
Examples: Glue ETL module, SageMaker processing module, EMR Spark module, Bedrock batch module

Module Configuration Examples 
modules:
  # Data cleaning module
  data_cleaning:
    executor: ${executors.glue_etl}
    entry_point: clean_data.py
    job_parameters:
      input_path: s3://bucket/raw/
      output_path: s3://bucket/cleaned/
    
  # Model training module
  model_training:
    executor: ${executors.python_processor}
    entry_point: train_model.py
    depends_on: [data_cleaning]  # Runs after data cleaning
Single Purpose: Each module handles one specific task
Composable: Can depend on other modules for execution order
Configurable: Parameterized through job parameters
Provider-Mapped: Module kind and executor determine the provider runtime

Executor

Module Runtime Template

What: Reusable runtime templates that define how modules run
When: Specify compute resources (instance types, frameworks) for your modules
Examples: SageMaker processors, Glue jobs, Spark clusters, Bedrock batch jobs

Executor Configuration Examples 
executors:
  # For Python-based data processing
  python_processor:
    class: sagemaker.sklearn.processing.SKLearnProcessor
    instance_type: ml.c5.2xlarge
    framework_version: 0.23-1
    
  # For big data processing
  glue_etl:
    type: glue_etl
    runtime: python3.9
    glue_version: "4.0"
    worker_type: G.2X
    number_of_workers: 5
Reusable: Multiple modules can use the same executor
Runtime Shape: Define compute resources and environments
Template-Based: Define once, use across many modules
Provider-Specific: Map neutral modules onto provider runtimes

Service

Long-Running Infrastructure

What: Long-running infrastructure components that the stack depends on
When: Deploy once per environment, then let applications and pipeline modules consume their outputs
Examples: SageMaker endpoints, Lambda APIs, DynamoDB tables, API Gateway, OpenSearch, Kinesis, EventBridge, Glue catalog resources

Service Configuration Example 
services:
  feature_store:
    type: dynamodb_table
    configuration:
      table_name: "ml-features-${parameters.environment}"
      partition_key: "feature_id"
      
  model_api:
    type: lambda_function
    configuration:
      function_name: "inference-api-${parameters.environment}"
      runtime: python3.9
      entry_point: "api.lambda_handler"
Stable: Once deployed, provide consistent interfaces
Shared: Pipeline modules and applications can use the same service outputs
Environment-Specific: Different configs for dev/staging/prod
Provider-Owned: Each provider controls service-specific deployment details

How Concepts Work Together

A stack is the boundary. Pipelines and services are deployed on different cadences inside that boundary.

Stack

Stack
System Boundary

Defines the environment, provider, services, modules, executors, parameters, and dependencies.

Pipeline Lifecycle

Pipeline
Workflow Graph
contains
Modules
Work Units
run on
Executors
Runtime Templates

Produces models, processed data, predictions, and other ML artifacts

Service Lifecycle

Services
Long-Running Infrastructure

Provides stable outputs such as table names, endpoint URLs, stream names, and API routes.

Complete Configuration Example 
name: ecommerce-recommendation-stack
version: v1.0

# Services: long-running infrastructure (mk s deploy)
services:
  feature_store:
    type: dynamodb_table
    configuration:
      table_name: "features-${parameters.environment}"

# Executors: reusable module runtime templates
executors:
  python_processor:
    class: sagemaker.sklearn.processing.SKLearnProcessor
    instance_type: ml.c5.xlarge
    
  glue_etl:
    type: glue_etl
    runtime: python3.9

# Modules: pipeline work units (mk p deploy)
modules:
  data_cleaning:
    executor: ${executors.glue_etl}     # Uses executor template
    entry_point: clean_data.py
    
  feature_engineering:
    executor: ${executors.python_processor}
    entry_point: build_features.py
    depends_on: [data_cleaning]         # Module dependency
    job_parameters:
      feature_table: "${services.feature_store.outputs.table_name}"  # Consumes service output

Deployment Model

Understanding when and how to deploy each concept:

Concept Command Frequency Purpose
Stack Compose file Defines the environment Top-level system boundary
Service mk s deploy Once per environment Stable infrastructure and outputs
Pipeline mk p deploy Multiple times per day Workflow orchestration
Module Part of pipeline Deployed with pipeline Individual work unit
Executor Configuration only Never deployed directly Runtime template for modules

Mental Model: Product Release System

Think of ModelKnife as the release system for an ML product:

  • Stack = the product boundary for one environment
  • Pipeline = the repeatable release workflow that creates artifacts
  • Module = one build, train, transform, or publish step in that workflow
  • Executor = the runtime template used to execute a module
  • Service = the long-running infrastructure that serves users or stores shared state

Key insight: pipelines change often as ML logic evolves; services change more carefully because they are stable integration points.

Common Patterns

1. Service-Pipeline Integration

Services provide stable interfaces that pipelines consume:

# Service provides feature storage
services:
  feature_store: { type: dynamodb_table }

# Pipeline modules read from and write to services
modules:
  model_training:
    job_parameters:
      feature_table: "${services.feature_store.outputs.table_name}"

2. Module Dependencies

Modules form a dependency graph that ModelKnife orchestrates:

modules:
  data_cleaning: { ... }
  feature_engineering:
    depends_on: [data_cleaning]
  model_training:
    depends_on: [feature_engineering]

3. Executor Reuse

Multiple modules can share executor configurations:

executors:
  ml_processor: { ... }  # Defined once
  
modules:
  feature_engineering:
    executor: ${executors.ml_processor}  # Reused
  model_training:
    executor: ${executors.ml_processor}  # Reused

Ready to Apply These Concepts?

Now that you understand ModelKnife's core concepts, try them out with real examples.

Quick Start Guide View Examples