Environment Variables and Secrets Management

Learn how to securely manage environment variables and secrets in your Trainwave jobs.

Quick Start


# Set a secret in your organization
wave secrets set OPENAI_API_KEY=sk-...


[env_vars]
OPENAI_API_KEY = "${OPENAI_API_KEY}"
BATCH_SIZE = "32"

Variable Types

Trainwave supports three types of variables:

Configuration Variables: Defined directly in trainwave.toml
Project Secrets: Scoped to a specific project
Organization Secrets: Available across all projects in your organization

Configuration Variables

Define variables directly in your trainwave.toml:


[env_vars]
# Fixed values
BATCH_SIZE = "32"
LEARNING_RATE = "0.001"
NUM_EPOCHS = "100"
 
# Local environment interpolation — references your local shell environment
WANDB_API_KEY = "${WANDB_API_KEY}"
HUGGINGFACE_TOKEN = "${HF_TOKEN}"

Secrets Management

Organization Secrets

Available to all projects in your organization:


# Set organization secrets
wave secrets set \
  OPENAI_API_KEY=sk-... \
  WANDB_API_KEY=abc...
 
# List organization secrets
wave secrets list

Project Secrets

Scoped to a specific project:


# Set project-specific secrets
wave secrets set \
  --project p-abc123 \
  DB_PASSWORD=xyz... \
  API_TOKEN=abc...
 
# List project secrets
wave secrets list --project p-abc123

Security Best Practices

Use Appropriate Scope

Set project-specific secrets at the project level — avoid storing them at the organization level unless they’re genuinely shared across all projects.

Rotate Secrets Regularly


# Update an existing secret with a new value
wave secrets set API_KEY=new-value

Common Patterns

Machine Learning Training


[env_vars]
# Authentication
WANDB_API_KEY = "${WANDB_API_KEY}"
HUGGINGFACE_TOKEN = "${HF_TOKEN}"
 
# Training parameters
BATCH_SIZE = "32"
LEARNING_RATE = "0.001"
NUM_EPOCHS = "100"
 
# Resource configuration
PYTORCH_CUDA_ALLOC_CONF = "max_split_size_mb:512"
OMP_NUM_THREADS = "4"

Distributed Training


[env_vars]
MASTER_ADDR = "localhost"
MASTER_PORT = "29500"
WORLD_SIZE = "4"
RANK = "0"
NCCL_DEBUG = "INFO"

Cloud Integration


[env_vars]
AWS_ACCESS_KEY_ID = "${AWS_ACCESS_KEY_ID}"
AWS_SECRET_ACCESS_KEY = "${AWS_SECRET_ACCESS_KEY}"
AWS_DEFAULT_REGION = "us-west-2"

Variable Precedence

Variables are resolved in the following order (highest to lowest priority):

Job-specific variables (in trainwave.toml)
Project secrets
Organization secrets
System environment variables

Troubleshooting

Missing Variable


# Check if the variable exists
wave secrets list | grep MISSING_VAR

Support

[email protected]