Operations Runbooks

This page contains step-by-step command sequences for manual AWS operations — running scripts directly from your machine with AWS credentials.

If you want to understand the automated pipeline (what happens when you push to main, how to trigger a deploy, how to promote to prod), see the CI/CD pipeline page instead.

Prerequisites

Install the following tools:

AWS CLI — configured with nebu-admin profile or equivalent
Terraform — version compatible with the modules
Docker — for building images
curl — for smoke tests

Verify your AWS identity:

aws sts get-caller-identity

Set the region if needed:

export AWS_REGION=us-west-1

Bootstrap Terraform state (first time only)

Run this once per environment before the very first Terraform plan. It creates the S3 bucket and DynamoDB table that Terraform uses to store and lock its state.

ENV=dev AWS_REGION=us-west-1 scripts/aws/bootstrap-terraform-state.sh

You only need to do this once. After this, the state bucket exists and all subsequent plans and applies will use it.

Deployment sequences

Deploy a backend code change

Use this when you've changed Python code, scripts, or Docker configuration but not the database schema:

scripts/aws/build-images.sh dev      # build new Docker images
scripts/aws/push-images.sh dev       # push to ECR
scripts/aws/deploy-services.sh dev   # force new ECS deployment
scripts/aws/smoke-test.sh dev        # verify the deployed API is healthy

Deploy a database schema change

Use this when you've added a migration file:

scripts/aws/terraform-plan.sh dev    # review infra changes (if any)
scripts/aws/terraform-apply.sh dev   # apply infra changes
scripts/aws/build-images.sh dev      # build new Docker images
scripts/aws/push-images.sh dev       # push to ECR
scripts/aws/run-migrations.sh dev    # run migrations BEFORE deploying new code
scripts/aws/deploy-services.sh dev   # force new ECS deployment
scripts/aws/smoke-test.sh dev        # verify

Run migrations before deploying new code

If new application code expects a database column that doesn't exist yet, it will crash on startup. Always run migrations first.

Deploy an infrastructure-only change

Use this when you've only changed Terraform files:

scripts/aws/terraform-plan.sh dev    # review what will change
scripts/aws/terraform-apply.sh dev   # apply changes
scripts/aws/smoke-test.sh dev        # verify

Deploy a frontend change

Use this when you've changed anything under frontend/. The frontend is a static React app — it is not a container. It is built locally, uploaded to an S3 bucket, and served by CloudFront.

Step 1 — Read the Vite environment variables from Terraform outputs:

FRONTEND_BUCKET=$(terraform -chdir=infra/terraform/envs/dev output -raw frontend_bucket_name)
CLOUDFRONT_ID=$(terraform -chdir=infra/terraform/envs/dev output -raw frontend_cloudfront_distribution_id)
COGNITO_USER_POOL_ID=$(terraform -chdir=infra/terraform/envs/dev output -raw cognito_user_pool_id)
COGNITO_CLIENT_ID=$(terraform -chdir=infra/terraform/envs/dev output -raw cognito_app_client_id)

Step 2 — Build the frontend with the correct environment variables:

VITE_AUTH_PROVIDER=cognito \
VITE_COGNITO_USER_POOL_ID="$COGNITO_USER_POOL_ID" \
VITE_COGNITO_CLIENT_ID="$COGNITO_CLIENT_ID" \
  npm run build --prefix frontend

Leave VITE_API_BASE_URL unset for the same-origin CloudFront deployment. The browser will call /v1/* relative to the CloudFront domain, which CloudFront routes to the API ALB automatically.

Step 3 — Upload the build to S3:

aws s3 sync frontend/dist "s3://$FRONTEND_BUCKET" --delete

The --delete flag removes files from S3 that no longer exist in frontend/dist/. Without it, old JavaScript bundles accumulate in the bucket.

Step 4 — Invalidate the CloudFront cache:

aws cloudfront create-invalidation \
  --distribution-id "$CLOUDFRONT_ID" \
  --paths "/*"

Without this step, users may see the old cached version for up to 24 hours. The invalidation takes 30–60 seconds to propagate. After it completes, a hard refresh in the browser loads the new version.

Verify: Open the frontend_base_url from Terraform outputs in a browser and confirm the new version is live.

terraform -chdir=infra/terraform/envs/dev output -raw frontend_base_url

For prod, substitute dev with prod

Use infra/terraform/envs/prod for all terraform output commands and use the prod S3 bucket and CloudFront distribution IDs.

Deploy a vega-core change

vega-core is a private Git submodule checked out under vega-core/. All backend containers are built from the repository root, so vega-core changes are included in the next normal image build. The runner image (vega-core-runner) is the primary consumer, but vega-api and vega-worker may also import from the submodule.

Step 1 — Update the submodule to the desired commit:

# Pull the latest commit on vega-core main
git -C vega-core pull origin main

# Or check out a specific commit
git -C vega-core checkout <commit-sha>

# Stage and commit the submodule pointer update
git add vega-core
git commit -m "chore: update vega-core to $(git -C vega-core rev-parse --short HEAD)"

Step 2 — Build and push new images (same as a backend deploy):

scripts/aws/build-images.sh dev
scripts/aws/push-images.sh dev

Step 3 — Apply Terraform to update the runner task definition:

The runner ECS task definition points to the vega-core-runner image by SHA tag. A Terraform apply is required to update it:

scripts/aws/terraform-plan.sh dev    # confirm only task definition changes
scripts/aws/terraform-apply.sh dev

Step 4 — Redeploy the services that use the updated images:

scripts/aws/deploy-services.sh dev
scripts/aws/smoke-test.sh dev

The runner task definition update takes effect for the next scan. Running scans are not interrupted.

Individual operations

Build Docker images

scripts/aws/build-images.sh dev

Builds all service images: - vega-api - vega-worker - vega-maintenance - vega-llm-proxy - vega-core-runner

Each image receives two tags: the current Git SHA (for traceability) and dev-current (for the ECS task definition to reference).

Push images to ECR

ECR (Elastic Container Registry) is AWS's Docker image registry. ECS pulls container images from ECR when starting tasks.

scripts/aws/push-images.sh dev

This script logs in to ECR, tags the local images with the full ECR repository URL, and pushes both the SHA and dev-current tags.

Deploy ECS services

scripts/aws/deploy-services.sh dev

Forces a new deployment for the vega-api, vega-worker, and vega-llm-proxy ECS services. ECS stops the old tasks and starts new ones using the current task definition (which points to dev-current images). The script waits for all services to report stable.

Run database migrations

scripts/aws/run-migrations.sh dev

Launches the vega-maintenance ECS task with the migration command, waits for the task to complete, and fails with a non-zero exit code if the container exits with an error.

For production:

scripts/aws/run-migrations.sh prod

Run smoke tests

scripts/aws/smoke-test.sh dev

Checks that the deployed API health endpoint returns 200. To override the URL:

API_BASE_URL=https://api.dev.vega.example.com scripts/aws/smoke-test.sh dev

If Terraform is installed, the script reads api_base_url from Terraform outputs automatically.

Manual AWS operations

Plan infrastructure changes

scripts/aws/terraform-plan.sh dev

Apply infrastructure changes

scripts/aws/terraform-apply.sh dev

# Production requires explicit confirmation:
scripts/aws/terraform-apply.sh prod --confirm-prod

Update a secret

AWS Secrets Manager stores encrypted secrets. Terraform creates the secret structure; you populate the value separately.

aws secretsmanager put-secret-value \
  --region us-west-1 \
  --secret-id <secret-id> \
  --secret-string '{"key":"value"}'

After updating a secret, redeploy the service that reads it so ECS picks up the new value:

scripts/aws/deploy-services.sh dev

Restart services without new images

Force ECS to replace running tasks with fresh tasks using the current task definition:

scripts/aws/deploy-services.sh dev

This is useful when you've updated a secret or environment variable in the task definition.

Check ECS service status

aws ecs describe-services \
  --region us-west-1 \
  --cluster $(terraform -chdir=infra/terraform/envs/dev output -raw ecs_cluster_name) \
  --services vega-api vega-worker vega-llm-proxy

Check SQS queue depth

aws sqs get-queue-attributes \
  --region us-west-1 \
  --queue-url <queue-url> \
  --attribute-names ApproximateNumberOfMessages ApproximateAgeOfOldestMessage

A large ApproximateAgeOfOldestMessage (in seconds) means scans are waiting too long — the worker may be down.

Tail CloudWatch logs

# Follow live logs for the API
aws logs tail /vega/dev/vega-api \
  --region us-west-1 \
  --since 10m \
  --follow

# View last hour of worker logs
aws logs tail /vega/dev/vega-worker \
  --region us-west-1 \
  --since 1h