Pipelines Reference

Complete reference for ML pipeline modules, executors, local development runs, and provider orchestration

Pipeline Components

Advanced Configuration

Available Executor Types

Data Processing

ML Compute

AI & Inference

Module Configuration

Individual processing units that make up your ML pipelines

Key Concept

Modules are the individual processing units that perform specific ML tasks. Each module uses an executor (compute environment) and contains your ML processing code.

Basic Module Structure 
modules:
  module_name:
    executor: ${executors.executor_name}
    repository: "../modules"
    source_dir: "jobs/training"
    # Optional compatibility alias for package root / legacy configs
    # code_path: "jobs/training"
    entry_point: "script.py"
    extra_py_files:
      - "./models:models"
      - "./configs:configs"
    additional_python_modules:
      - "pandas==2.2.3"
    job_parameters:
      input_path: "s3://bucket/input/"
      output_path: "s3://bucket/output/"
      custom_param: "value"
    depends_on: ["other_module"]
    description: "Description of what this module does"

Module Parameters

  • executor - Reference to executor configuration (required)
  • repository - Path to module code repository
  • source_dir - Optional package root inside the repository. SageMaker and local Python run from this staged package root.
  • code_path - Optional package-root alias used by shared Python packaging. Prefer source_dir for new configs.
  • entry_point - Main script or class to execute. For Python packages, this is relative to source_dir.
  • extra_py_files - Optional files or directories copied into the staged package. Use source:target to control the import path.
  • submit_py_files - Executor-specific Python files submitted alongside the main job, retained for EMR compatibility.
  • files - Runtime assets localized for executors that support file distribution, such as EMR and Glue.
  • additional_python_modules - Optional Python requirements added to the staged package requirements file.
  • inputs and outputs - Named pipeline data mappings used by compatible executors.
  • input_names and output_names - Name or list of names that identify which job_parameters contain local/SageMaker input and output paths.
  • job_parameters - Run parameters passed to your code. Processing-style executors pass them as CLI arguments; SageMaker Training converts them to hyperparameters.
  • depends_on - List of modules that must complete first
  • description - Human-readable description

Module Dependencies and Execution Order

Pipeline with Module Dependencies 
modules:
  data_ingestion:
    executor: ${executors.glue_etl}
    entry_point: "ingest_data.py"
    job_parameters:
      source_database: "raw_data"
      output_path: "s3://bucket/ingested/"
    # No dependencies - runs first
    
  data_cleaning:
    executor: ${executors.python_processor}
    entry_point: "clean_data.py"
    job_parameters:
      input_path: "s3://bucket/ingested/"
      output_path: "s3://bucket/cleaned/"
    depends_on: ["data_ingestion"]
    
  feature_engineering:
    executor: ${executors.python_processor}
    entry_point: "create_features.py"
    job_parameters:
      input_path: "s3://bucket/cleaned/"
      output_path: "s3://bucket/features/"
    depends_on: ["data_cleaning"]
    
  model_training:
    executor: ${executors.training_job}
    entry_point: "train_model.py"
    job_parameters:
      training_data: "s3://bucket/features/"
      model_output: "s3://bucket/models/"
      epochs: 50
    depends_on: ["feature_engineering"]

Runtime Parameters

Deploy-time defaults with run-time overrides for pipeline modules

Execution Model

ModelKnife resolves project parameters into each module's job_parameters when the stack is loaded. During orchestration, modules read from module_parameters.<module>, so scheduled runs have defaults and manual runs can override only the values that need to change.

Default module parameters 
parameters:
  start_date: "2026-06-06"
  training_data_path: "s3://example/train/${parameters.start_date}/"

modules:
  fit_preprocessor:
    executor: ${executors.processor}
    entry_point: fit_preprocessor_job.py
    job_parameters:
      input: ${parameters.training_data_path}
      start-date: ${parameters.start_date}
      output: "s3://example/preprocessor/${runtime.run_id}/"

Run Overrides

Manual run overrides 
mk p run -p train --param start_date=2026-06-01
mk p run -p train --module-param fit_preprocessor.fit-batches=64
  • --param KEY=VALUE overrides a pipeline-level parameter, then ModelKnife materializes affected module job_parameters before the run starts.
  • --module-param MODULE.KEY=VALUE overrides one module parameter directly.
  • ${runtime.run_id} and ${runtime.started_at} are resolved by the orchestration runtime so all modules in the same run share the same context.
  • AWS Step Functions receives only the materialized module_parameters context; users do not need to model provider-specific runtime input by hand.

Profiles and Workspaces

Separate configuration inheritance from deployment identity

A profile selects and merges compose files; a workspace isolates the deployed stack and state. Use profiles for broad variants such as development versus production, and workspace_overrides for small values that differ among workspaces using the same profile.

Workspace-specific pipeline value 
parameters:
  output_bucket: shared-dev-output

workspace_overrides:
  alice:
    parameters:
      output_bucket: alice-dev-output
  ci:
    parameters:
      output_bucket: ci-dev-output

modules:
  export_features:
    executor: ${executors.processor}
    entry_point: export_features.py
    job_parameters:
      output: "s3://${parameters.output_bucket}/${runtime.run_id}/"
Select both dimensions explicitly 
mk p deploy --profile dev --workspace alice
mk p run --profile dev --workspace alice
mk p status --profile dev --workspace alice
  • Workspace overrides are applied after profile inheritance and before placeholder resolution.
  • The override key must match the resolved workspace from --workspace, compose workspace, or the current-user default.
  • ${system.workspace} is available at compose resolution time. Runtime placeholders include ${runtime.run_id}, ${runtime.started_at}, and ${runtime.pipeline_name}.
  • See the YAML reference for the full merge order and supported top-level keys.

Executor Configuration Overrides

Override executor parameters at the module level for fine-tuned resource allocation

Key Concept

You can override specific executor parameters at the module level while still using the base executor configuration. This is useful when you need different compute resources or settings for specific modules.

Module-level executor overrides 
executors:
  python_processor:
    type: sagemaker_processor
    class: sagemaker.sklearn.processing.SKLearnProcessor
    instance_type: "ml.m5.large"  # Base configuration
    instance_count: 1
    framework_version: "1.0-1"

modules:
  # Uses base executor configuration
  data_cleaning:
    executor: ${executors.python_processor}
    entry_point: "clean_data.py"
    
  # Override instance type for memory-intensive task
  feature_engineering:
    executor: ${executors.python_processor}
    entry_point: "create_features.py"
    instance_type: "ml.m5.2xlarge"  # Override: bigger instance
    instance_count: 2               # Override: more instances
    
  # Override for GPU processing
  model_training:
    executor: ${executors.python_processor}
    entry_point: "train_model.py"
    instance_type: "ml.p3.2xlarge"  # Override: GPU instance
    max_runtime_in_seconds: 7200    # Override: longer timeout

Override Benefits

  • Cost Optimization: Use smaller instances for lightweight tasks, larger for heavy computation
  • Performance Tuning: Adjust timeouts, memory, and CPU for specific workloads
  • Resource Management: Scale compute resources per module without duplicating executor definitions
  • Maintainability: Keep base configurations simple while customizing where needed

Executor Types

Compute environments that run your modules

Local Python

Run one module locally for development validation before deploying to a cloud executor

Best For

Fast local checks for imports, packaging, argument parsing, and small sample runs. Local Python is not a production scheduler and is not deployed by mk p deploy.

Execution Model

ModelKnife stages source_dir, entry_point, extra_py_files, and additional_python_modules into .mlknife/build/local-python, then runs the entry point from that staged package. This intentionally catches missing package files before the same module is sent to SageMaker.

Executor Configuration 
executors:
  local_python:
    type: local_python
    environment:
      manager: venv
      path: .venv
      python: python3
      create: false
      install: false

Environment Parameters

  • manager - Local environment manager. Phase 1 supports venv by default; non-venv values use the configured Python command directly.
  • path - Virtual environment path relative to the module repository. Defaults to .venv.
  • python - Python command used to create the venv or run the module. Defaults to python3.
  • create - Create the venv when it is missing. Defaults to false.
  • install - Install configured requirements and staged additional_python_modules. Defaults to false.
  • requirements - Optional list of requirements files to install when install: true.

Module Usage

Local Module Configuration 
modules:
  fit_preprocessor:
    executor: ${executors.local_python}
    repository: .
    source_dir: jobs/training
    entry_point: fit_preprocessor_job.py
    extra_py_files:
      - ./models:models
      - ./configs:configs
    additional_python_modules:
      - pandas==2.2.3
      - pyarrow==18.1.0
    input_names: input
    output_names: output
    job_parameters:
      input: ./samples/train.parquet
      output: ./outputs/preprocessor
      max-vocab-size: 100000

Run Command

Single-module Local Run 
mk p run -p train --local --module fit_preprocessor
  • Phase 1 runs a single module only. Full local DAG execution is intentionally deferred.
  • job_parameters are passed through transparently as --key value arguments.
  • input_names and output_names use the same field names as SageMaker Processing. For local runs, named local filesystem values are resolved to absolute paths from the compose file directory; URI values such as S3 paths are preserved.
  • For named outputs, ModelKnife creates the parent directory before running the script.
  • Unnamed relative parameters are passed through as written. Your script owns any custom path semantics outside input_names and output_names.
  • Logs and command metadata are written under .mlknife/runs.
  • local_python is run-only. mk p deploy rejects modules that use it so local validation cannot accidentally become a production workflow.

EMR Spark Jar

Serverless Spark jobs using EMR Serverless for cost-effective big data processing

Best For

Large-scale Scala/Java Spark applications, complex ETL with custom JARs, cost-effective big data processing with automatic scaling, and batch processing workloads.

EMR Serverless Benefits

  • Cost Optimization: Pay only for compute used, automatic start/stop
  • Serverless: No cluster management, automatic scaling
  • Performance: Optimized Spark runtime with faster startup
  • Integration: Native AWS service integration
Configuration Example 
executors:
  emr_spark_processor:
    type: emr_spark

    # Optional settings. Omit application_id to use the shared application.
    application_id: "00abc123example"
    executor_num: 4                      # Number of executor instances
    executor_cores: 2
    executor_memory: "4g"
    executor_disk: "20g"                 # Disk space per executor
    driver_cores: 2
    driver_memory: "4g"
    driver_disk: "20g"                   # Disk space for driver
    timeout: 1440                        # 24 hours timeout (in minutes)

    # Optional: All parameters below can be customized
    dependent_jars:                      # Executor-level dependencies
      - "org.apache.spark:spark-sql_2.12:3.3.0"
    config:
      - spark.sql.adaptive.enabled=true
      - spark.sql.adaptive.coalescePartitions.enabled=true
      - spark.serializer=org.apache.spark.serializer.KryoSerializer

Configuration Parameters

  • type - Use emr_spark for Scala/Java Spark applications
  • role_arn - IAM execution role ARN (optional, uses default EMR execution role)
  • application_id - Custom EMR Serverless application ID (optional, uses shared application by default)
  • executor_num - Number of executor instances (optional, default: 4)
  • executor_cores - Number of CPU cores per executor (optional, default: 2)
  • executor_memory - Memory per executor (optional, default: 4g)
  • executor_disk - Disk space per executor (optional, default: 20g)
  • driver_cores - Number of CPU cores for driver (optional, default: 2)
  • driver_memory - Memory for driver (optional, default: 4g)
  • driver_disk - Disk space for driver (optional, default: 20g)
  • timeout - Job timeout in minutes (optional, default: 720 = 12 hours)
  • dependent_jars - List of JAR files or Maven packages for executor-level dependencies
  • config - List of Spark configuration parameters

Module Usage

Note: All executor-level parameters can be overridden at the module level for fine-grained control per job.

Module Configuration 
modules:
  data_transformation:
    executor: ${executors.emr_spark_processor}
    repository: "../spark-app"
    entry_point: "com.company.DataProcessor"
    build_command: "mvn clean package -DskipTests"
    # Override executor-level settings for this specific job
    executor_num: 8                      # Override executor instances
    executor_memory: "8g"                # Override memory for large job
    timeout: 1440                        # Override timeout to 24 hours
    job_parameters:
      input_path: "s3://data-bucket/input/"
      output_path: "s3://data-bucket/output/"
      config_param: "production"
    submit_jar: "target/data-processor-1.0.jar"
    dependent_jars:                      # Override/extend executor JARs
      # Local JAR files (uploaded to S3)
      - "lib/custom-library.jar"
      # Maven packages (downloaded automatically)
      - "org.apache.spark:spark-sql_2.12:3.3.0"
      - "com.databricks:spark-xml_2.12:0.14.0"
      # S3 JAR files
      - "s3://shared-jars/delta-core_2.12-2.3.0.jar"

Module Configuration Parameters

  • executor - Reference to the configured EMR Spark executor
  • repository - Local path to directory containing Spark application project
  • entry_point - Main class name to execute (must have main method)
  • build_command - Shell command to build the project (e.g., Maven, SBT)
  • Executor Overrides - Any executor parameter can be overridden at module level:
    • executor_num, executor_cores, executor_memory, executor_disk
    • driver_cores, driver_memory, driver_disk
    • timeout, application_id, role_arn
    • dependent_jars, config
  • job_parameters - Custom parameters passed to your Spark application as arguments:
    • input_path - S3 path for input data to process
    • output_path - S3 path where processed data will be written
    • Custom parameters passed as --key value arguments
  • submit_jar - Path to the main JAR file (relative to repository, auto-detected if not specified)
  • dependent_jars - List of JAR files or Maven packages (overrides executor-level):
    • JAR files: Local paths or S3 URIs (added to spark.jars)
    • Maven packages: Format groupId:artifactId:version (added to spark.jars.packages)

EMR Application Management

Shared Application (Default):

  • Uses team-shared EMR Serverless application created during mk setup init
  • Cost-efficient for typical workloads
  • Automatic start/stop management
  • No configuration required

Custom Application:

  • Specify application_id for dedicated EMR application
  • Useful for resource isolation, different environments, or custom configurations
  • Application must exist before job submission
  • Full control over application lifecycle and settings

EMR PySpark

Serverless PySpark jobs using EMR Serverless for Python-based big data processing

Best For

Large-scale Python data processing, ETL pipelines with Python libraries, machine learning feature engineering, and cost-effective distributed computing for Python workloads.

Configuration Example 
executors:
  emr_pyspark_processor:
    type: emr_pyspark

    # Optional settings. Omit application_id to use the shared application.
    application_id: "00def456uvw789123"  # Custom application ID
    executor_num: 4                      # Number of executor instances
    executor_cores: 2
    executor_memory: "4g"
    executor_disk: "20g"                 # Disk space per executor
    driver_cores: 2
    driver_memory: "4g"
    driver_disk: "20g"                   # Disk space for driver
    timeout: 720                         # 12 hours timeout (in minutes)

    # Optional: All parameters below can be customized
    dependent_jars:                      # Executor-level JAR dependencies
      - "io.delta:delta-core_2.12:2.3.0"
    extra_py_files:                      # Executor-level Python files
      - "src/utils:utils"               # add utils package to the root 
      - "common/helpers.py"
    files:                               # Runtime files localized for the driver
      - "configs/feature_flags.json"
    additional_python_modules:           # Auto-built Python packages
      - delta-spark==2.4.0
      - s3fs==2025.5.1
      - numpy==1.24.4
    # Optional isolated environment; Docker builds and uploads the archive.
    python_environment:
      requirements:
        - pandas==2.2.3
        - pyarrow==18.1.0
      alias: python_env
      python: ./python_env/bin/python
    config:
      - spark.sql.adaptive.enabled=true
      - spark.sql.adaptive.coalescePartitions.enabled=true
      - spark.python.worker.memory=2g

Configuration Parameters

  • type - Use emr_pyspark for Python Spark applications
  • role_arn - IAM execution role ARN (optional, uses default EMR execution role)
  • application_id - Custom EMR Serverless application ID (optional, uses shared application by default)
  • executor_num - Number of executor instances (optional, default: 4)
  • executor_cores - Number of CPU cores per executor (optional, default: 2)
  • executor_memory - Memory per executor (optional, default: 4g)
  • executor_disk - Disk space per executor (optional, default: 20g)
  • driver_cores - Number of CPU cores for driver (optional, default: 2)
  • driver_memory - Memory for driver (optional, default: 4g)
  • driver_disk - Disk space for driver (optional, default: 20g)
  • timeout - Job timeout in minutes (optional, default: 720 = 12 hours)
  • dependent_jars - List of JAR files or Maven packages for executor-level dependencies
  • extra_py_files - List of additional Python files/directories for Python import distribution
  • files - Runtime files such as JSON/YAML configuration assets. AWS EMR uploads local files to S3 and passes them through spark.files.
  • additional_python_modules - List of Python packages to install and build automatically
  • python_environment - Mapping that defines an EMR-compatible Python environment. Use requirements (or modules) to build an archive with Docker, or archive for a prebuilt archive. Optional keys include pip_options, docker_image, docker_platform, alias, build_python, and python.
  • python_env_archive - Compatibility shortcut for a prebuilt environment archive URI; may include a #alias.
  • python_env_python - Python executable inside the localized environment archive, for example ./python_env/bin/python.
  • config - List of Spark configuration parameters

Module Usage

Note: All executor-level parameters can be overridden at the module level for fine-grained control per job.

Module Configuration 
modules:
  feature_engineering:
    executor: ${executors.emr_pyspark_processor}
    repository: "../pyspark_jobs"
    entry_point: "feature_engineering.py"
    # Override executor-level settings for this specific job
    executor_num: 6                      # Override executor instances
    executor_memory: "8g"                # Override memory for large dataset
    driver_memory: "4g"                  # Override driver memory
    timeout: 360                         # Override timeout to 6 hours
    job_parameters:
      input_path: "s3://data-bucket/raw/"
      output_path: "s3://data-bucket/features/"
      feature_columns: "age,income,location"
      target_column: "conversion"
    submit_py_files:
      - "utils/data_helpers.py"
      - "transformers/"
    dependent_jars:                      # Override/extend executor JARs
      # Maven packages (downloaded automatically)
      - "io.delta:delta-core_2.12:2.3.0"
      - "org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.3.0"
      # S3 JAR files
      - "s3://jars-bucket/custom-transformers.jar"
    extra_py_files:                      # Override/extend executor Python files
      - "module_specific_utils/"
    files:                               # Runtime config files for the driver
      - "configs/feature_flags.json"
    additional_python_modules:           # Add module-specific packages
      - scikit-learn==1.3.0
      - matplotlib==3.7.1

Module Configuration Parameters

  • executor - Reference to the configured EMR PySpark executor
  • repository - Local path to directory containing PySpark job scripts
  • entry_point - Main PySpark script to execute (becomes spark-submit application)
  • files - Runtime files localized by Spark. Read them from the driver with pyspark.SparkFiles.get("filename").
  • job_parameters - Custom parameters passed to your PySpark script as arguments:
    • input_path - S3 path for input data to process
    • output_path - S3 path where processed data will be written
    • Custom parameters passed as --key value arguments
  • submit_py_files - List of additional Python files/directories to include in PySpark job (zipped and distributed to executors)
  • dependent_jars - List of JAR files or Maven packages:
    • JAR files: Local paths or S3 URIs (e.g., Delta Lake, Iceberg JARs)
    • Maven packages: Format groupId:artifactId:version for automatic download

Cost Optimization

EMR PySpark uses the same shared EMR Serverless application as EMR Spark, providing:

  • Zero idle costs: Application stops after 15 minutes of inactivity
  • Fast startup: Applications start automatically when jobs are submitted
  • Shared resources: Team members share the same EMR application
  • Automatic scaling: Executors scale based on workload

Glue Job

ETL jobs using AWS Glue for large-scale data processing with Delta Lake support

Best For

Large-scale ETL operations, data transformation, Spark-based processing with automatic scaling, Delta Lake operations, and serverless execution with built-in data lake capabilities.

Delta Lake Integration

All Glue jobs come pre-configured with Delta Lake support, including:

  • Automatic Delta Core: io.delta:delta-core_2.12:2.3.0 JAR included by default
  • Delta Catalog: Spark SQL Delta catalog configured automatically
  • S3 LogStore: Optimized Delta log store for S3 storage
  • Iceberg Support: AWS Glue catalog integration for Iceberg tables
Configuration Example 
executors:
  etl_processor:
    type: glue_etl

    # Optional: All parameters below show defaults - can be customized
    role_arn: "mlknife-glue-job"
    runtime: "python3.9"
    glue_version: "5.0"
    executor_type: "G.1X"
    executor_num: 2
    timeout: 2880  # 48 hours in minutes

    # Optional: All parameters below can be customized
    dependent_jars:                      # Executor-level JAR dependencies
      - "io.delta:delta-core_2.12:2.3.0"
    extra_py_files:                      # Executor-level Python files
      - "src/utils:utils"               # add utils package to the root 
      - "common/helpers.py"
    files:                               # Runtime files localized for the job
      - "configs/feature_flags.json"
    additional_python_modules:           # Auto-built Python packages
      - delta-spark==2.4.0
      - s3fs==2025.5.1
      - numpy==1.24.4
    config:
      - spark.sql.adaptive.enabled=true
      - spark.sql.adaptive.coalescePartitions.enabled=true
      - spark.python.worker.memory=2g

Configuration Parameters

  • role_arn - IAM role ARN for Glue job execution (defaults to mlknife-glue-job)
  • runtime - Python runtime version (default: "python3.9")
  • glue_version - AWS Glue version (default: "5.0")
  • executor_type - Worker node type: G.1X, G.2X, G.4X, G.8X (default: "G.1X")
  • executor_num - Number of worker nodes (default: 2)
  • number_of_workers, worker_type, and role - Deprecated compatibility names. Prefer executor_num, executor_type, and role_arn.
  • timeout - Job timeout in minutes (default: 2880 = 48 hours)
  • dependent_jars - Additional JAR files or Maven coordinates (default: includes Delta Lake)
  • extra_py_files - Additional Python files to include for Python imports
  • files - Runtime files such as JSON/YAML configuration assets. AWS Glue uploads local files to S3 and passes them through --extra-files.
  • additional_python_modules - Python packages to install
  • config - Custom Spark configuration overrides

Default Spark Configuration

Every Glue job includes optimized Spark settings:

  • Hive Integration: AWS Glue Data Catalog as Hive metastore
  • Delta Lake: Delta catalog and S3 log store configured
  • Iceberg Support: Glue catalog integration for Iceberg tables
  • Arrow Optimization: PyArrow enabled with 500 records per batch
  • Performance: 300 shuffle partitions, dynamic partition overwrite
  • Memory: 2GB Python worker memory allocation
  • Monitoring: Metrics and Spark UI enabled by default

Module Usage

Module Configuration 
modules:
  etl_transformation:
    executor: "${executors.etl_processor}"
    repository: "../glue_jobs"
    entry_point: "transform_data.py"
    job_parameters:
      input_path: "s3://data-bucket/raw/"
      output_path: "s3://data-bucket/processed/"

Module Configuration Parameters

  • executor - Reference to the configured Glue job executor
  • repository - Local path to directory containing Glue job scripts
  • entry_point - Main Python script to execute as Glue job
  • Executor Overrides - Any executor parameter can be overridden at module level:
    • runtime, glue_version, worker_type, number_of_workers
    • timeout, role_arn
    • dependent_jars, extra_py_files, additional_python_modules
    • config - Custom Spark configurations
  • job_parameters - Custom parameters passed to your Glue job script as job arguments:
    • input_path - S3 path for source data to process
    • output_path - S3 path where processed data will be written
    • Custom parameters passed as --key value arguments
  • dependent_jars - List of JAR files or Maven packages (overrides executor-level):
    • Maven coordinates: "org.apache.spark:spark-avro_2.12:3.3.0"
    • S3 URIs: "s3://bucket/path/to/custom.jar"
    • Note: Delta Lake JAR is included automatically

SageMaker PySpark Processor

Distributed data processing using PySpark on SageMaker

Best For

Large-scale feature engineering with Python, distributed data transformations, complex aggregations and joins, and ML data preprocessing pipelines.

Configuration Example 
executors:
  spark_processor:
    type: sagemaker_processor
    class: sagemaker.spark.processing.PySparkProcessor
    instance_type: "ml.m5.xlarge"
    instance_count: 2
    max_runtime_in_seconds: 3600
    spark_config:
      spark.executor.memory: "4g"
      spark.executor.cores: "2"
      spark.sql.adaptive.enabled: "true"

Configuration Parameters

  • class - Use sagemaker.spark.processing.PySparkProcessor
  • instance_type - EC2 instance type for Spark cluster
  • instance_count - Number of instances in the cluster
  • spark_config - Spark configuration parameters
  • max_runtime_in_seconds - Maximum job runtime
  • role_arn - Optional SageMaker execution role ARN. If omitted, ModelKnife resolves the configured sagemaker_execution_role.

Module Usage

Module Configuration 
modules:
  feature_engineering:
    executor: "${executors.spark_processor}"
    repository: "../spark_jobs"
    entry_point: "feature_engineering.py"
    job_parameters:
      input_path: "s3://data-bucket/raw/"
      output_path: "s3://data-bucket/processed/"
    submit_py_files:
      - "utils/spark_helpers.py"
      - "transformers/"
    additional_python_modules:
      - "delta-spark==2.4.0"

Module Configuration Parameters

  • executor - Reference to the configured SageMaker PySpark processor executor
  • repository - Local path to directory containing Spark job scripts
  • entry_point - Main PySpark script to execute (becomes spark-submit application)
  • job_parameters - Custom parameters passed to your Spark script as arguments:
    • input_path - S3 path for input data to process
    • output_path - S3 path where processed data will be written
  • submit_py_files - List of additional Python files/directories to include in Spark job (zipped and distributed to executors)
  • additional_python_modules - List of Python packages to download as wheels and include in Spark job

SageMaker Spark JAR Processor

Run Java/Scala Spark applications on SageMaker

Best For

High-performance Scala/Java Spark applications, complex business logic in JVM languages, integration with existing Java/Scala codebases, and performance-critical data processing.

Configuration Example 
executors:
  spark_jar_processor:
    type: sagemaker_processor
    class: sagemaker.spark.processing.SparkJarProcessor
    instance_type: "ml.m5.2xlarge"
    instance_count: 3
    max_runtime_in_seconds: 7200
    spark_config:
      spark.executor.memory: "8g"
      spark.executor.cores: "4"
      spark.serializer: "org.apache.spark.serializer.KryoSerializer"

Configuration Parameters

  • class - Use sagemaker.spark.processing.SparkJarProcessor
  • instance_type - EC2 instance type for Spark cluster
  • instance_count - Number of instances in the cluster
  • spark_config - Spark configuration parameters
  • max_runtime_in_seconds - Maximum job runtime
  • role_arn - Optional SageMaker execution role ARN. If omitted, ModelKnife resolves the configured sagemaker_execution_role.

Module Usage

Module Configuration 
modules:
  data_transformation:
    executor: "${executors.spark_jar_processor}"
    repository: "../spark-app"
    entry_point: "com.company.DataProcessor"
    build_command: "mvn clean package -DskipTests"
    job_parameters:
      input_path: "s3://data-bucket/input/"
      output_path: "s3://data-bucket/output/"
    submit_jar: "target/data-processor-1.0.jar"
    dependent_jars:
      - "lib/custom-library.jar"

Module Configuration Parameters

  • executor - Reference to the configured SageMaker Spark JAR processor executor
  • repository - Local path to directory containing Spark application project
  • entry_point - Main class name to execute (must have main method)
  • build_command - Shell command to build the project (e.g., Maven, SBT)
  • job_parameters - Custom parameters passed to your Spark application as arguments:
    • input_path - S3 path for input data to process
    • output_path - S3 path where processed data will be written
  • submit_jar - Path to the main JAR file (relative to repository, auto-detected if not specified)
  • dependent_jars - List of additional JAR files to include in Spark classpath

Build Process

The Spark JAR processor automatically builds your project using the specified build_command before execution. Ensure your project structure follows Maven or SBT conventions.

SageMaker Processor

Managed Python processing jobs with ModelKnife source packaging

Best For

Data preprocessing, feature engineering, model evaluation, and custom Python processing where the same module package should be validated locally and then run on SageMaker.

Executor Configuration 
executors:
  data_processor:
    type: sagemaker_processor
    class: sagemaker.pytorch.processing.PyTorchProcessor
    instance_type: "ml.m5.xlarge"
    instance_count: 1
    framework_version: "2.4.0"
    py_version: "py311"
    volume_size_in_gb: 50
    max_runtime_in_seconds: 3600
    # Optional. If omitted, ModelKnife resolves the configured SageMaker role.
    role_arn: "arn:aws:iam::123456789012:role/mlknife-sagemaker"

Configuration Parameters

  • class - SageMaker processor class to use, such as sagemaker.pytorch.processing.PyTorchProcessor, sagemaker.sklearn.processing.SKLearnProcessor, or another ScriptProcessor subclass.
  • instance_type - EC2 instance type for processing.
  • instance_count - Number of processing instances.
  • framework_version and py_version - Framework runtime for framework processors.
  • volume_size_in_gb - EBS volume size for the processing job.
  • max_runtime_in_seconds - Maximum job runtime.
  • role_arn - Optional SageMaker execution role ARN. If omitted, ModelKnife resolves the configured sagemaker_execution_role.

Module Usage

Packaged Processing Module 
modules:
  data_preprocessing:
    executor: ${executors.data_processor}
    repository: .
    source_dir: jobs/processing
    entry_point: preprocess_data.py
    extra_py_files:
      - ./models:models
      - ./configs:configs
    additional_python_modules:
      - pandas==2.2.3
      - pyarrow==18.1.0
    input_names: raw_data
    output_names: processed_data
    job_parameters:
      raw_data: "s3://data-bucket/raw/"
      processed_data: "s3://data-bucket/processed/"
      feature-config: configs/features.json

Module Configuration Parameters

  • executor - Reference to the configured SageMaker processor executor.
  • repository - Local path to the code repository.
  • source_dir - Package root used when ModelKnife stages a SageMaker source bundle. The entry point should be inside this directory.
  • code_path - Optional package-root alias accepted by the shared Python packaging path. Prefer source_dir for new modules.
  • entry_point - Main Python script to execute for processing, relative to source_dir.
  • extra_py_files - Extra files or directories copied into the staged source bundle. Use source:target to keep clean import paths.
  • additional_python_modules - Python requirements merged into the staged requirements.txt and installed by the SageMaker SDK run script.
  • input_names - Comma-separated input channel names. Each name must match a job_parameters key. ModelKnife creates ProcessingInput objects and passes local container paths to your script.
  • output_names - Comma-separated output channel names. Each name must match a job_parameters key. ModelKnife creates ProcessingOutput objects and passes local container paths to your script.
  • job_parameters - Parameters passed to your processing script as --key value arguments. Input and output channel keys are replaced with their SageMaker container paths.

Packaging Behavior

When code_path, extra_py_files, or additional_python_modules is set, ModelKnife stages a source package before building the SageMaker pipeline. source_dir selects the package root used for that staged source. In verbose deploy output, the package root, staged source directory, rewritten entry point, and dependency list are shown for inspection.

SageMaker Training

Framework estimator training with packaged source and hyperparameter-based script arguments

Best For

GPU or CPU model training with SageMaker framework estimators such as PyTorch. Use this when SageMaker should own the training job lifecycle and your script reads inputs or channels and writes model artifacts.

Executor Configuration 
executors:
  model_trainer:
    type: sagemaker_training
    class: sagemaker.pytorch.PyTorch
    instance_type: "ml.g4dn.xlarge"
    instance_count: 1
    framework_version: "2.4.0"
    py_version: "py311"
    volume_size_in_gb: 125
    max_runtime_in_seconds: 14400
    hyperparameters:
      batch-size: 512
      learning-rate: 0.001

Configuration Parameters

  • class - SageMaker estimator class to use, such as sagemaker.pytorch.PyTorch or another EstimatorBase subclass.
  • instance_type - EC2 instance type for training.
  • instance_count - Number of training instances.
  • framework_version and py_version - Framework runtime for framework estimators.
  • volume_size_in_gb - Training volume size.
  • max_runtime_in_seconds - Maximum training runtime.
  • hyperparameters - Default estimator hyperparameters. Module job_parameters are merged into these at deploy time.
  • role_arn - Optional SageMaker execution role ARN. If omitted, ModelKnife resolves the configured sagemaker_execution_role.

Module Usage

Packaged Training Module 
modules:
  model_training:
    executor: ${executors.model_trainer}
    repository: .
    source_dir: jobs/training
    entry_point: train_model.py
    inputs:
      training: "s3://data-bucket/train/"
      validation: "s3://data-bucket/validation/"
    extra_py_files:
      - ./models:models
      - ./configs:configs
    additional_python_modules:
      - pandas==2.2.3
      - pyarrow==18.1.0
    job_parameters:
      model-store-path: "s3://model-bucket/artifacts/"
      epochs: 10
      batch-size: 1024

Module Configuration Parameters

  • executor - Reference to the configured SageMaker training executor.
  • repository - Local path to the code repository.
  • source_dir - Package root used when ModelKnife stages a SageMaker source bundle.
  • code_path - Optional package-root alias accepted by the shared Python packaging path. Prefer source_dir for new modules.
  • entry_point - Main Python script to execute for training, relative to source_dir.
  • extra_py_files - Extra files or directories copied into the staged source bundle.
  • additional_python_modules - Python requirements merged into the staged requirements.txt.
  • inputs - Optional SageMaker training input channels. Each key becomes a training channel and each value is wrapped as a SageMaker TrainingInput when needed.
  • job_parameters - Merged into estimator hyperparameters. SageMaker framework estimators pass hyperparameters to your entry point as script arguments.

Training Parameters

SageMaker Training does not use Processing-style input_names or output_names. Use estimator inputs for mounted training channels, and use job_parameters for script arguments and hyperparameter overrides. When inputs is a map of S3 URIs, ModelKnife wraps each value as a SageMaker TrainingInput.

Bedrock Batch Inference (Managed)

Queues jobs to a management service (Lambda + DynamoDB) that calls Bedrock on your behalf

Best For

Large-scale batch inference with foundation models, processing thousands of prompts efficiently using AWS Bedrock's batch inference capabilities.

Why This Executor?

This is not the native Bedrock batch API wrapper. It integrates with a queue-based management service (Lambdas + DynamoDB) to provide throttling, retries, scheduling, and cost/concurrency control for large batch runs. Use it when you need operational safeguards around Bedrock at scale.

How It Differs From Native

  • Queue + Control: Jobs are queued; concurrency is capped to avoid quota errors.
  • Resilience: Managed retries/status checks via Step Functions and Lambdas.
  • Separation: Decouples submission from execution; safer for spikes.
  • Observability: Status stored in DynamoDB; periodic polling (wait_seconds).
Configuration Example 
executors:
  bedrock_batch_infer:
    # Supported provider-neutral alias: foundation_batch_infer
    type: bedrock_batch_infer
    service_name: bedrock  # optional; defaults to 'bedrock'
    # Lambda ARNs for the batch manager service (configure per env)
    queue_lambda_arn: arn:aws:lambda:REGION:ACCOUNT:function:bedrock-batch-inference-create-batch-queue
    process_lambda_arn: arn:aws:lambda:REGION:ACCOUNT:function:bedrock-batch-inference-process-batch-job
    status_lambda_arn: arn:aws:lambda:REGION:ACCOUNT:function:bedrock_batch_inference_status_checker
    # Optional: polling wait between status checks (seconds)
    wait_seconds: 300

How Bedrock Batch Works

  • Enqueue: A Lambda creates/queues batch jobs in a DynamoDB-backed queue
  • Process: A Lambda starts Bedrock batch jobs and updates job state
  • Status: A Lambda checks progress; Step Functions polls every wait_seconds
  • Results: Outputs are written to the configured S3 directory

Required Parameters (in modules)

  • task_name - Unique identifier for the batch job
  • prompt_input_dir - S3 path containing input prompts (JSONL format)
  • result_output_dir - S3 path for batch inference results
  • model_id - Bedrock foundation model identifier (e.g., Claude, Titan)

Module Usage

Module Configuration 
modules:
  llm_batch_processing:
    executor: "${executors.bedrock_batch_infer}"
    job_parameters:
      task_name: "content-analysis-batch"
      prompt_input_dir: "s3://data-bucket/prompts/"
      result_output_dir: "s3://data-bucket/results/"
      model_id: "anthropic.claude-3-sonnet-20240229-v1:0"
    description: "Process large batches of prompts using Bedrock"
    depends_on: []

Module Configuration Parameters

  • executor - Reference to the configured Bedrock batch inference executor
  • job_parameters - Required parameters for Bedrock batch inference:
    • task_name - Unique identifier for the batch inference job
    • prompt_input_dir - S3 path containing input prompts in JSONL format
    • result_output_dir - S3 path where batch inference results will be written
    • model_id - Amazon Bedrock foundation model identifier (e.g., Claude, Titan models)
  • description - Human-readable description of the batch processing task
  • depends_on - List of modules that must complete before this batch job runs

Service Dependencies

This executor requires the following AWS resources (configured via the executor):

  • queue_lambda_arn: ARN of the queue-creation Lambda
  • process_lambda_arn: ARN of the batch-processing Lambda
  • status_lambda_arn: ARN of the status-checker Lambda
  • IAM roles for Bedrock and S3 access