Skip to content

CI/CD Pipeline

This page explains the automated CI/CD pipeline for Vega — what runs automatically, what requires a manual trigger, and how to perform or roll back a deployment.

Overview

There are three GitHub Actions workflows in .github/workflows/:

Workflow File Trigger What it does
CI ci.yml Every PR and every push to main Runs tests, validates Terraform, checks frontend build
Deploy Dev deploy-dev.yml Auto after CI passes on main, or manual Builds images, scans, pushes to ECR, deploys to dev
Promote Prod promote-prod.yml Manual only (requires a dev-validated SHA) Promotes an immutable image tag from dev ECR to prod

The normal flow is:

Open PR → CI runs → merge to main → CI runs again → Deploy Dev auto-triggers → dev is updated
                                                              ↓
                                               (validate dev manually)
                                                              ↓
                                              Trigger Promote Prod with the SHA
                                                              ↓
                                                      prod is updated

Workflow: CI

Runs on: every pull request and every push to main.

Job What it checks
backend-tests Runs pytest tests with all Python dependencies
frontend-build Runs npm ci && npm run build to verify the frontend compiles
terraform-validate Checks terraform fmt, init, and validate for both dev and prod environments
shell-syntax Checks bash -n syntax on all scripts/aws/*.sh files

If any job fails, the PR is blocked and Deploy Dev will not trigger.

Workflow: Deploy Dev

Runs automatically after CI succeeds on main. Can also be triggered manually from the GitHub Actions tab.

The workflow assumes the GitHub Actions deploy role (AWS_DEV_DEPLOY_ROLE_ARN) via OIDC — no long-lived AWS credentials are stored in GitHub.

Steps in order

  1. Checkout — checks out the commit with submodules (including vega-core/) using VEGA_CORE_CHECKOUT_TOKEN
  2. Resolve release SHA — captures the 12-character Git SHA that will become the immutable image tag
  3. Configure AWS credentials — assumes the vega-dev-github-actions-deploy IAM role via GitHub OIDC
  4. Build service images — runs scripts/aws/build-images.sh dev <sha>, which builds five Docker images: vega-api, vega-worker, vega-maintenance, vega-llm-proxy, vega-core-runner
  5. Scan images with Trivy — fails the workflow if any image has an unfixed HIGH or CRITICAL vulnerability
  6. Generate SBOMs — generates a CycloneDX SBOM for each image with Syft and uploads them as artifacts
  7. Push images to ECR — runs scripts/aws/push-images.sh dev <sha>, which pushes each image with both the immutable SHA tag and the dev-current mutable tag
  8. Terraform plan dev — runs scripts/aws/terraform-plan.sh dev
  9. Reject unexpected destroys — fails the workflow if the plan includes any destructive changes on resource types that shouldn't be destroyed (excluding ECS task definitions, which rotate normally)
  10. Upload Terraform plan — saves the plan as an artifact for review
  11. Terraform apply dev — runs scripts/aws/terraform-apply.sh dev
  12. Run dev migrations — runs scripts/aws/run-migrations.sh dev <sha>, which launches the vega-maintenance ECS task to apply pending Alembic migrations
  13. Deploy dev ECS services — runs scripts/aws/deploy-services.sh dev <sha>, which force-deploys vega-api, vega-worker, and vega-llm-proxy and waits for them to stabilize
  14. Smoke test dev — runs scripts/aws/smoke-test.sh dev to verify the API health endpoint returns 200

Frontend is not deployed automatically

The CI/CD pipeline does not currently upload the frontend to S3. Frontend deployments are manual. See Deploy a frontend change.

Workflow: Promote Prod

Runs only when manually triggered from the GitHub Actions tab. It requires you to input:

  • image_sha — the immutable 12-character Git SHA that was already built, scanned, pushed to ECR, and smoked in dev. This prevents building new code directly against prod.
  • deploy_worker_path — set to true only when prod worker/LLM proxy credentials are populated and scan execution has been approved.

The workflow assumes the vega-prod-github-actions-deploy IAM role. It runs in the prod GitHub Environment, which can be configured with required reviewers to add a manual approval gate before the workflow proceeds.

Steps in order

  1. Checkout — checks out current main with submodules
  2. Resolve and validate release SHA — rejects dev-current or prod-current as input (only accepts an immutable SHA)
  3. Configure AWS credentials — assumes the prod deploy role via OIDC
  4. Verify image tags exist in ECR — confirms the provided SHA tag exists in ECR for all five image repositories before proceeding
  5. Terraform plan prod — generates a prod plan
  6. Reject unexpected prod destroys — same guard as dev, but applied to prod
  7. Upload prod Terraform plan — saves the plan for audit
  8. Terraform apply prod — applies the plan with the --confirm-prod safety flag
  9. Run prod migrations — applies pending migrations against the prod database
  10. Deploy prod ECS services — force-deploys prod services and waits for stability
  11. Smoke test prod — verifies the prod API health endpoint returns 200

Triggering a manual dev deploy

To re-deploy dev without merging new code (for example, after updating a secret or Terraform configuration):

  1. Go to Actions → Deploy Dev in GitHub
  2. Click Run workflow on the main branch
  3. The workflow runs exactly the same steps as the automatic trigger

Promoting a SHA to production

  1. Find the SHA to promote. It must already have passed the full Deploy Dev run: 

    # The SHA is the 12-character prefix of the commit that triggered Deploy Dev
git log --oneline -10
    Or look at the "Resolve release SHA" step output in a successful Deploy Dev run in GitHub Actions.

  2. Go to Actions → Promote Prod in GitHub

  3. Click Run workflow and enter:

    • image_sha: the 12-character SHA (e.g., abc123def456)
    • deploy_worker_path: false unless explicitly enabling scan execution in prod

  4. If the prod GitHub Environment has required reviewers configured, approve the deployment when prompted.

  5. Monitor the run. The workflow uploads the Terraform plan as an artifact — download and review it before the apply step if the plan includes unexpected changes.

Rollback procedure

Because images are tagged with immutable Git SHAs, rolling back means re-promoting the last known-good SHA.

Backend rollback (API, worker, LLM proxy, runner):

  1. Find the previous good SHA from a successful Deploy Dev or Promote Prod run
  2. Trigger Promote Prod with that SHA

This re-applies the Terraform task definitions pointing to the old images and force-deploys ECS services to pull them.

Manual rollback (dev only):

PREV_SHA=<previous-12-char-sha>

# Update Terraform variables to point to the old SHA
TF_VAR_api_image_tag=$PREV_SHA \
TF_VAR_runner_image_tag=$PREV_SHA \
TF_VAR_llm_proxy_image_tag=$PREV_SHA \
scripts/aws/terraform-apply.sh dev

scripts/aws/deploy-services.sh dev "$PREV_SHA"
scripts/aws/smoke-test.sh dev

Frontend rollback:

The previous frontend bundle is not automatically retained. To roll back the frontend, check out the previous commit, rebuild with the same Vite environment variables, and re-upload to S3. See Deploy a frontend change.

Database migrations are not automatically reversed

Rolling back application code does not undo database migrations. If the previous code cannot run against the current schema, you must apply a down-migration or a corrective migration manually. Design migrations to be backward-compatible with the previous application version to make rollbacks safe.

Required GitHub configuration

The workflows require the following to be set on the repository:

Type Name Value
Repository variable AWS_REGION us-west-1
Repository variable AWS_DEV_DEPLOY_ROLE_ARN ARN of the vega-dev-github-actions-deploy IAM role (from terraform output github_actions_dev_role_arn)
Repository variable AWS_PROD_DEPLOY_ROLE_ARN ARN of the vega-prod-github-actions-deploy IAM role (from terraform output github_actions_prod_role_arn)
Repository secret VEGA_CORE_CHECKOUT_TOKEN Fine-grained PAT with read access to vega-backend and vega-core (for private submodule checkout)
GitHub Environment dev No required reviewers needed
GitHub Environment prod Add required reviewers for a manual approval gate

The deploy roles are defined in Terraform: - infra/terraform/envs/dev/ci_role.tf - infra/terraform/envs/prod/ci_role.tf