Team Access Management

Set up role-based access control for your ModelKnife team using AWS IAM groups and permissions

Overview

ModelKnife provides team-based access control through AWS IAM groups. This allows you to manage who can deploy services, run pipelines, and perform administrative tasks.

Two-Tier Access Model

ModelKnife uses a simple two-tier access model designed for most ML teams:

  • Developers: Can deploy and manage ML pipelines and services
  • Administrators: Can manage team membership and AWS resources

Quick Navigation

This guide covers the complete setup process:

Prerequisites

Before running mk setup init, ensure your AWS account and IAM user are properly configured with the necessary permissions.

Important

The first admin user who runs mk setup init needs comprehensive IAM permissions to create AWS resources and manage team access. This is a one-time setup requirement.

1. AWS Account Setup Verification

Verify your AWS account is ready for ModelKnife setup:

# Check your AWS connection
aws sts get-caller-identity

Expected output should show your account ID, user ARN, and user ID:

{
    "UserId": "AIDACKCEVSQ6C2EXAMPLE",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/your-username"
}

2. AWS Console Setup Instructions

This section provides detailed step-by-step instructions for setting up IAM permissions through the AWS Management Console. Follow these steps if you prefer using the web interface over CLI commands.

Security Best Practice

Do not use your AWS root account for ModelKnife setup. Create a dedicated IAM user with administrative permissions instead. The steps below will guide you through creating this user.

Two Setup Options

You can set up IAM permissions using either:

  • Option A: AWS Managed Policy (AdministratorAccess) - Quick but very broad permissions
  • Option B (Recommended): Custom Policy - Minimum required permissions for better security

Step 1: Access the AWS IAM Console

  1. Sign in to the AWS Management Console
  2. Navigate to the IAM service by:
    • Typing "IAM" in the search bar at the top
    • Or clicking "Services" → "Security, Identity, & Compliance" → "IAM"
  3. You should see the IAM dashboard with navigation options on the left

Step 2: Create IAM User (If Needed)

If you don't already have an IAM user for ModelKnife administration:

  1. Navigate to Users:
    • Click "Users" in the left navigation panel
    • Click the "Add users" button
  2. Configure User Details:
    • Enter username: modelknife-admin (or your preferred name)
    • Check both access types:
      • ✅ "Programmatic access" (for AWS CLI)
      • ✅ "AWS Management Console access" (for web interface)
    • For console access, choose "Custom password" and set a secure password
    • Uncheck "Require password reset" if you don't want to change it on first login
    • Click "Next: Permissions"
  3. Set Permissions (Choose One Option):
    Option A: AWS Managed Policy (Quick Setup)
    • Click "Attach existing policies directly"
    • Search for "AdministratorAccess"
    • Check the box next to "AdministratorAccess"
    • Click "Next: Tags"
    Note: This provides very broad permissions but may cause errors in some regions
  4. Add Tags (Optional):
    • Add tags if your organization requires them
    • Example: Key="Purpose", Value="ModelKnife Administration"
    • Click "Next: Review"
  5. Review and Create:
    • Review all settings
    • Click "Create user"
    • Important: Download the CSV file with credentials or copy the Access Key ID and Secret Access Key
    • Store these credentials securely - you won't be able to see the secret key again

Step 3: Create Custom Policy (Recommended Option)

If you chose Option B above, follow these detailed steps to create the custom policy:

  1. Navigate to Policies:
    • In the IAM console, click "Policies" in the left navigation
    • Click the "Create policy" button
  2. Choose JSON Editor:
    • Click the "JSON" tab (not the Visual editor)
    • You'll see a default policy template
  3. Replace with ModelKnife Policy:
    • Delete all existing content in the JSON editor
    • Copy the complete policy from the code block in section 5 below
    • Paste it into the JSON editor
    • The policy should start with {"Version": "2012-10-17"...
  4. Validate and Continue:
    • Click "Next: Tags" (AWS will validate the JSON automatically)
    • If you see any JSON errors, double-check that you copied the complete policy
  5. Add Tags (Optional):
    • Add tags if required by your organization
    • Click "Next: Review"
  6. Name and Create Policy:
    • Policy name: ModelKnifeAdminPolicy
    • Description: Minimum required permissions for ModelKnife setup and administration
    • Review the policy summary to ensure it includes IAM permissions
    • Click "Create policy"

Step 4: Attach Policy to Existing User

If you already have an IAM user and need to add ModelKnife permissions:

  1. Navigate to Your User:
    • Click "Users" in the left navigation
    • Click on your username from the list
  2. Add Permissions:
    • Click the "Permissions" tab
    • Click "Add permissions" button
    • Select "Attach existing policies directly"
  3. Select Policy:
    • For AWS managed policy: Search for "AdministratorAccess" and check it
    • For custom policy: Search for "ModelKnifeAdminPolicy" and check it
    • Click "Next: Review"
  4. Review and Apply:
    • Review the permissions being added
    • Click "Add permissions"

Step 5: Generate Access Keys (If Needed)

If your user doesn't have programmatic access keys for AWS CLI:

  1. Access Security Credentials:
    • In your user's details page, click the "Security credentials" tab
    • Scroll down to "Access keys" section
  2. Create Access Key:
    • Click "Create access key"
    • Select "Command Line Interface (CLI)" as the use case
    • Check the confirmation checkbox
    • Click "Next"
  3. Add Description (Optional):
    • Add a description like "ModelKnife CLI access"
    • Click "Create access key"
  4. Save Credentials:
    • Important: Copy both the Access Key ID and Secret Access Key
    • Or click "Download .csv file" to save them
    • Store these securely - you won't be able to see the secret key again
    • Click "Done"

Step 6: Verify Console Setup

After completing the console setup, verify everything is configured correctly:

Test your setup with these commands:

# Test AWS connection
aws sts get-caller-identity

# Test IAM permissions
aws iam get-user

# Test policy listing (should not give permission errors)
aws iam list-policies --max-items 1

Console Setup Troubleshooting

Error: "Policy document is invalid"

Cause: JSON syntax error in custom policy

Solution:

  • Ensure you copied the complete policy JSON from section 5
  • Check for missing commas, brackets, or quotes
  • Use a JSON validator to check syntax
  • Try copying the policy again from the source
Error: "Cannot find policy ModelKnifeAdminPolicy"

Cause: Custom policy not created or named differently

Solution:

  • Go to IAM → Policies and verify the policy exists
  • Check the exact policy name (case-sensitive)
  • If missing, recreate the policy following the Step 3 instructions above
  • Refresh the user permissions page
Error: "Access keys not working with AWS CLI"

Cause: Incorrect AWS CLI configuration

Solution:

  • Run aws configure and re-enter credentials
  • Verify access key ID and secret key are correct
  • Check that the user has programmatic access enabled
  • Ensure no extra spaces in the credentials
Error: "User already exists" when creating user

Cause: Username already taken in your AWS account

Solution:

  • Choose a different username (e.g., "mk-admin", "modelknife-setup")
  • Or use an existing user and just attach the policy
  • Check if the existing user already has the required permissions

Console Setup Tips

  • Browser Compatibility: Use a modern browser (Chrome, Firefox, Safari, Edge)
  • Multiple Tabs: Keep the policy creation tab open while creating users
  • Credential Security: Never share access keys or store them in code repositories
  • Policy Testing: Use AWS IAM Policy Simulator to test permissions before applying
  • Documentation: Keep a record of the username and policy name for future reference

Security Best Practices

  • Unique Usernames: Use descriptive names like "modelknife-admin" instead of generic names
  • Strong Passwords: Use complex passwords for console access
  • Access Key Rotation: Regularly rotate access keys (every 90 days recommended)
  • Principle of Least Privilege: Use the custom policy instead of AdministratorAccess when possible
  • MFA Recommendation: Consider enabling Multi-Factor Authentication for admin users

3. AWS CLI Configuration

Configure AWS CLI with your IAM user credentials:

# Configure AWS CLI
aws configure

# Or configure a specific profile
aws configure --profile modelknife-admin

You'll need to provide:

  • AWS Access Key ID: From your IAM user
  • AWS Secret Access Key: From your IAM user
  • Default region: Your preferred AWS region (e.g., us-east-1)
  • Default output format: json (recommended)

4. Required IAM Permissions

The admin user needs comprehensive IAM permissions to run mk setup init. This section provides the complete list of required permissions organized by category, along with explanations for why each permission is needed.

Principle of Least Privilege

These permissions follow the principle of least privilege - they provide exactly what's needed for ModelKnife setup, no more. The setup process creates service-specific roles that handle AWS service permissions, so your admin user only needs IAM management capabilities.

Complete IAM Permissions Policy

Use this complete policy for your admin user. You can either copy the JSON below or download the ready-to-use policy file:

You can download the policy file directly to use in the AWS Console:

Download Policy JSON

Or copy and paste this JSON directly into a custom IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "IAMRoleManagement",
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:GetRole",
                "iam:AttachRolePolicy",
                "iam:DetachRolePolicy",
                "iam:PutRolePolicy",
                "iam:GetRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:ListAttachedRolePolicies",
                "iam:ListRolePolicies",
                "iam:PassRole"
            ],
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "iam:RoleName": "mlknife-*"
                }
            }
        },
        {
            "Sid": "IAMGroupManagement",
            "Effect": "Allow",
            "Action": [
                "iam:CreateGroup",
                "iam:GetGroup",
                "iam:AttachGroupPolicy",
                "iam:DetachGroupPolicy",
                "iam:AddUserToGroup",
                "iam:RemoveUserFromGroup",
                "iam:ListGroupsForUser"
            ],
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "iam:GroupName": "mlknife-*"
                }
            }
        },
        {
            "Sid": "IAMPolicyManagement",
            "Effect": "Allow",
            "Action": [
                "iam:CreatePolicy",
                "iam:GetPolicy",
                "iam:GetPolicyVersion"
            ],
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "iam:PolicyName": "MLKnife*"
                }
            }
        },
        {
            "Sid": "IAMReadOnlyOperations",
            "Effect": "Allow",
            "Action": [
                "iam:GetUser",
                "iam:GetAccountSummary",
                "iam:ListUsers",
                "iam:ListGroups",
                "iam:ListPolicies",
                "iam:ListRoles"
            ],
            "Resource": "*"
        },
        {
            "Sid": "STSOperations",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Permission Categories Explained

IAM Role Management

Why needed: ModelKnife creates 9 service-specific IAM roles for AWS services (SageMaker, Lambda, Glue, Step Functions, etc.)

  • iam:CreateRole - Create new service roles
  • iam:GetRole - Check if roles already exist
  • iam:AttachRolePolicy - Attach AWS managed policies to roles
  • iam:PutRolePolicy - Add custom inline policies to roles
  • iam:PassRole - Allow services to assume the created roles

Resources created:

  • mlknife-stepfunctions-{region}-{profile} - Step Functions execution role
  • mlknife-eventbridge-{region}-{profile} - EventBridge scheduling role
  • mlknife-lambda-execution-{region}-{profile} - Lambda function execution role
  • mlknife-sagemaker-endpoint-{region}-{profile} - SageMaker endpoint role
  • mlknife-feature-store-{region}-{profile} - Feature Store access role
  • mlknife-api-gateway-{region}-{profile} - API Gateway integration role
  • mlknife-search-service-{region}-{profile} - OpenSearch Serverless role
  • mlknife-sagemaker-exec-{region}-{profile} - SageMaker processing/training role
  • mlknife-glue-job-{region}-{profile} - AWS Glue job execution role
IAM Group Management

Why needed: ModelKnife uses IAM groups for team access control

  • iam:CreateGroup - Create developer and admin groups
  • iam:GetGroup - Check group membership and status
  • iam:AttachGroupPolicy - Attach policies to groups
  • iam:AddUserToGroup - Add team members to appropriate groups

Groups created:

  • mlknife-developers - Standard team member access
  • mlknife-admin - Administrative access for team management
IAM Policy Management

Why needed: ModelKnife creates custom policies for team access control

  • iam:CreatePolicy - Create custom policies for groups
  • iam:GetPolicy - Check if policies already exist
  • iam:GetPolicyVersion - Verify policy content

Policies created:

  • MLKnifeDeveloperPolicy - Permissions for developers to deploy services and pipelines
  • MLKnifeAdminPolicy - Additional permissions for team management
IAM Read-Only Operations

Why needed: For setup validation and user management

  • iam:GetUser - Verify user exists before adding to groups
  • iam:ListUsers - Display team members
  • sts:GetCallerIdentity - Identify current user and AWS account

Alternative: Using AWS Managed Policies

If you prefer to use AWS managed policies instead of the custom policy above, you can attach these policies to your admin user:

# Option 1: Full administrative access (broadest permissions)
arn:aws:iam::aws:policy/AdministratorAccess

# Option 2: IAM-specific administrative access (more restrictive)
arn:aws:iam::aws:policy/IAMFullAccess

How to Use the Copied Policy

After copying the policy JSON above, follow these steps in the AWS Console:

  1. Go to AWS IAM Policies
  2. Click "Create policy"
  3. Click the "JSON" tab
  4. Paste the copied policy JSON
  5. Click "Next: Tags" → "Next: Review"
  6. Name it "ModelKnifeAdminPolicy"
  7. Click "Create policy"
  8. Attach the policy to your IAM user

Managed Policy Limitations

AdministratorAccess: Very broad permissions but may cause "Invalid principal" errors in some AWS regions/accounts due to service principal restrictions.

IAMFullAccess: Provides all IAM permissions but lacks STS permissions needed for account identification.

Recommendation: Use the custom policy above for the most reliable setup experience.

Additional Policy Examples

For specific organizational requirements, here are additional policy examples you can use:

Minimal IAM-Only Policy

If your organization requires the absolute minimum permissions, use this IAM-only policy. Note that this may require additional permissions during setup:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:GetRole",
                "iam:AttachRolePolicy",
                "iam:PutRolePolicy",
                "iam:PassRole",
                "iam:CreateGroup",
                "iam:GetGroup",
                "iam:AttachGroupPolicy",
                "iam:CreatePolicy",
                "iam:GetPolicy",
                "iam:GetUser",
                "iam:AddUserToGroup",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}
Organization-Scoped Policy

For organizations that want to restrict ModelKnife resources to specific naming patterns:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:GetRole",
                "iam:AttachRolePolicy",
                "iam:PutRolePolicy",
                "iam:PassRole"
            ],
            "Resource": "arn:aws:iam::*:role/mlknife-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateGroup",
                "iam:GetGroup",
                "iam:AttachGroupPolicy"
            ],
            "Resource": "arn:aws:iam::*:group/mlknife-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreatePolicy",
                "iam:GetPolicy"
            ],
            "Resource": "arn:aws:iam::*:policy/MLKnife*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetUser",
                "iam:AddUserToGroup",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

What mk setup init Creates For You

The setup command automatically creates all AWS service permissions and roles. You don't need to manually configure:

  • 9 IAM service roles (SageMaker, Lambda, Glue, Step Functions, etc.)
  • Service-specific policies for S3, DynamoDB, OpenSearch, etc.
  • Team access groups with appropriate permissions
  • All role trust relationships and policy attachments

You only need IAM management permissions - the setup handles everything else automatically.

Why Only IAM Permissions?

The setup process is designed to be simple - you only provide IAM management permissions, and ModelKnife creates everything else:

  • You provide: IAM permissions to create roles, groups, and policies
  • Setup creates: All service roles with appropriate AWS service permissions
  • Result: Team members get permissions through groups, not direct user policies

This approach is more secure and manageable than giving broad permissions to individual users.

6. Permission Verification Steps

Before running mk setup init, verify that your IAM permissions are correctly configured using these validation commands:

Basic AWS Access Verification

# 1. Verify AWS connection and identity
aws sts get-caller-identity

# 2. Check your current IAM user details
aws iam get-user

# 3. Verify AWS region configuration
aws configure get region

IAM Permission Validation

Test specific IAM permissions required for ModelKnife setup:

# Test role management permissions
aws iam list-roles --max-items 1

# Test group management permissions  
aws iam list-groups --max-items 1

# Test policy management permissions
aws iam list-policies --scope Local --max-items 1

# Test user management permissions
aws iam list-users --max-items 1

Advanced Permission Testing

For more thorough validation, test actual resource creation (these commands create and immediately delete test resources):

Optional Advanced Testing

These commands create temporary resources to test permissions. They're optional but provide the most accurate validation.

# Test role creation (creates and deletes a test role)
aws iam create-role --role-name mlknife-test-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "lambda.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }' && aws iam delete-role --role-name mlknife-test-role

# Test group creation (creates and deletes a test group)
aws iam create-group --group-name mlknife-test-group && \
aws iam delete-group --group-name mlknife-test-group

Expected Verification Results

All verification commands should complete successfully:

aws sts get-caller-identity

Returns your AWS account ID, user ARN, and user ID

aws iam get-user

Returns your IAM user details without permission errors

aws iam list-roles

Returns list of existing IAM roles (proves read access)

aws iam list-groups

Returns list of existing IAM groups

Role/Group Creation Tests

Successfully creates and deletes test resources

Troubleshooting Permission Issues

Error: "AccessDenied" on list operations

Cause: Missing basic IAM read permissions

Solution:

  • Ensure your IAM user has the custom policy from section 5
  • Or attach IAMReadOnlyAccess managed policy as minimum
  • Contact your AWS administrator to grant permissions
Error: "User not found" or "Invalid credentials"

Cause: AWS CLI not configured or invalid credentials

Solution:

  • Run aws configure to set up credentials
  • Check credentials: aws configure list
  • Verify access keys are active in IAM console
Error: "Cannot create role" during advanced testing

Cause: Missing role creation permissions

Solution:

  • Verify the custom policy includes all IAM role permissions
  • Check that the policy is attached to your user account
  • Ensure you're using the correct AWS profile/credentials
  • Try the minimal policy validation commands first

Policy Validation Commands

Use these commands to validate your policy is working correctly before running mk setup init:

# Validate policy JSON syntax (if you have the file locally)
aws iam validate-policy-document --policy-document file://modelknife-admin-policy.json

# Test specific IAM permissions
aws iam simulate-principal-policy \
    --policy-source-arn $(aws sts get-caller-identity --query Arn --output text) \
    --action-names iam:CreateRole iam:CreateGroup iam:CreatePolicy \
    --resource-arns "*"

# Quick permission check - should return without errors
aws iam get-account-summary --query 'SummaryMap.{Users:Users,Groups:Groups,Roles:Roles}' --output table

Copy-Paste Policy Verification

After copying and pasting the policy JSON, verify it was applied correctly:

# List policies attached to your user
aws iam list-attached-user-policies --user-name $(aws sts get-caller-identity --query 'Arn' --output text | cut -d'/' -f2)

# Get the policy document to verify it matches
aws iam get-policy --policy-arn arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):policy/ModelKnifeAdminPolicy \
    --query 'Policy.{PolicyName:PolicyName,Description:Description}' --output table

# Test the policy allows basic operations
aws iam list-roles --max-items 1 --query 'Roles[0].RoleName' --output text
  • Ensure you have iam:CreateRole and iam:DeleteRole permissions
  • Check if there are resource-based restrictions in your policy
  • Try using the complete custom policy from section 5
  • Automated Permission Verification

    For convenience, here are comprehensive validation commands that test all required permissions:

    # Comprehensive permission validation script
    # Save this as validate-permissions.sh and run: bash validate-permissions.sh
    
    #!/bin/bash
    echo "=== ModelKnife IAM Permission Validation ==="
    echo "Testing all required permissions for mk setup init..."
    echo
    
    # Test 1: Basic AWS access
    echo "1. Testing AWS connection..."
    if aws sts get-caller-identity > /dev/null 2>&1; then
        echo "   ✅ AWS connection successful"
        USER_ARN=$(aws sts get-caller-identity --query 'Arn' --output text)
        echo "   User: $USER_ARN"
    else
        echo "   ❌ AWS connection failed"
        exit 1
    fi
    
    # Test 2: IAM read permissions
    echo "2. Testing IAM read permissions..."
    aws iam get-user > /dev/null 2>&1 && echo "   ✅ IAM user read access" || echo "   ❌ IAM user read failed"
    aws iam list-roles --max-items 1 > /dev/null 2>&1 && echo "   ✅ IAM role list access" || echo "   ❌ IAM role list failed"
    aws iam list-groups --max-items 1 > /dev/null 2>&1 && echo "   ✅ IAM group list access" || echo "   ❌ IAM group list failed"
    
    # Test 3: IAM creation permissions (creates and deletes test resources)
    echo "3. Testing IAM creation permissions..."
    if aws iam create-role --role-name mlknife-test-role --assume-role-policy-document '{
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {"Service": "lambda.amazonaws.com"},
            "Action": "sts:AssumeRole"
        }]
    }' > /dev/null 2>&1; then
        echo "   ✅ IAM role creation successful"
        aws iam delete-role --role-name mlknife-test-role > /dev/null 2>&1
        echo "   ✅ IAM role deletion successful"
    else
        echo "   ❌ IAM role creation failed"
    fi
    
    if aws iam create-group --group-name mlknife-test-group > /dev/null 2>&1; then
        echo "   ✅ IAM group creation successful"
        aws iam delete-group --group-name mlknife-test-group > /dev/null 2>&1
        echo "   ✅ IAM group deletion successful"
    else
        echo "   ❌ IAM group creation failed"
    fi
    
    echo
    echo "=== Validation Complete ==="
    echo "If all tests show ✅, you're ready to run 'mk setup init'"

    Or run individual validation commands:

    # Quick validation commands
    aws sts get-caller-identity  # Should return your user info
    aws iam get-user            # Should return your IAM user details
    aws iam list-roles --max-items 1  # Should list roles without errors
    aws iam list-groups --max-items 1 # Should list groups without errors
    
    # Test policy attachment (using a harmless AWS managed policy)
    aws iam attach-user-policy --user-name YOUR_USERNAME --policy-arn arn:aws:iam::aws:policy/IAMReadOnlyAccess
    aws iam detach-user-policy --user-name YOUR_USERNAME --policy-arn arn:aws:iam::aws:policy/IAMReadOnlyAccess

    This script will test all permissions and provide a detailed report of what works and what needs to be fixed.

    Permission Validation Tips

    • Start Simple: Begin with basic verification commands before advanced testing
    • Check Policies: Use AWS IAM console to review attached policies
    • Test Incrementally: If one command fails, fix that before proceeding
    • Use Simulation: AWS IAM Policy Simulator can test permissions without creating resources