Skip to content

Hardening and Maintenance

Hardening is the set of protections that keep the backend stable when things go wrong: users submit oversized files, scans hang indefinitely, workers crash, or someone tries to run more scans than the system can handle.

Quotas and limits

ScanCapacityPolicy (application/scans/) evaluates quotas before allowing scan creation. Key limits:

Setting Default What it guards
VEGA_MAX_ACTIVE_SCANS_PER_USER 4 Prevents one user from monopolizing scan capacity
VEGA_MAX_SOURCE_BYTES 2147483648 (2 GB) Maximum source archive size at upload time
VEGA_MAX_ARCHIVE_ENTRIES 50,000 Maximum files in an uploaded archive
VEGA_MAX_ARCHIVE_UNCOMPRESSED_BYTES 5368709120 (5 GB) Maximum total uncompressed archive size

These are checked at the API or use-case layer before any work begins. Requests that exceed limits get a 4xx error with a quota_exceeded code and a human-readable message.

The scan capacity configuration (ScanCapacityPolicy) also controls ECS parallelism:

Field Default Description
max_workers_per_runner 4 Maximum audit workers (components) per ECS runner task
max_fast_runners_per_scan 1 Maximum parallel fast-mode runner tasks per scan
max_verification_runners_per_scan 0 Maximum parallel verification runner tasks (0 = disabled)

Worker heartbeats

Workers send regular heartbeats via POST /api/ops/workers/heartbeat. The CloudWatchWorkerHeartbeatAdapter (or InMemoryWorkerHeartbeatAdapter locally) tracks:

  • Which workers are registered
  • When each worker last sent a heartbeat
  • Whether any workers have gone stale (heartbeat older than VEGA_WORKER_HEARTBEAT_TTL_SECONDS, default 120 seconds)

Operators can inspect worker state via GET /api/ops/workers. A stale worker indicates the process has crashed and needs to be restarted.

Stale scan recovery

In the scan worker's polling loop, before processing new scan messages, ReconcileScansUseCase checks for scans that have been running longer than VEGA_SCAN_RUNNING_STALE_SECONDS (default: 6 hours).

Stale scans happen when: - A runner task crashes without writing a failure event - The ECS task is stopped by AWS before finishing - A network partition prevents the runner from writing its final status

The reconcile use case: 1. Detects scans in running state older than the stale threshold 2. Checks if the associated ECS runner task is still alive 3. If the ECS task is gone, marks the scan failed with a stale-scan error event 4. Preserves any partial findings and events already written

Similarly, ReconcileRepositoryIngestUseCase recovers stale IngestJob records in the ingest worker.

Cleanup

OperationsMaintenanceUseCase (application/operations/) handles removal of stale records:

  • Expired planning artifacts (past their expires_at timestamp)
  • Old worker heartbeat registrations
  • Orphaned session records
  • Artifact retention enforcement per VEGA_ARTIFACT_RETENTION_DAYS

Billing eligibility

Before creating a scan, ScanEligibilityUseCase (application/billing_policy/) checks:

  • User has sufficient balance (via BillingPort)
  • No active spend limit is exceeded
  • A scan key can be provisioned via ScanKeyPolicy

If eligibility fails, the scan is rejected with a billing_insufficient_funds or quota_exceeded error before any resources are consumed.

The /api/ops/ endpoints

api/routers/operations.py exposes operational state through protected API endpoints. These require operator or root role.

Endpoint What it returns / does
GET /api/ops/limits Current quota settings
POST /api/ops/limits/evaluate Evaluate whether a specific operation would be within limits
POST /api/ops/workers/heartbeat Worker heartbeat registration
GET /api/ops/workers Worker heartbeat registry
POST /api/ops/cleanup Trigger stale artifact cleanup
POST /api/ops/maintenance Trigger maintenance tasks

Maintenance task

The vega-maintenance ECS task runs scripts/run-maintenance.py. It is a one-off task (runs and exits). The most common use is database migrations:

scripts/aws/run-migrations.sh dev

Other maintenance operations (cleanup, artifact pruning) can be triggered by overriding the task command or via POST /api/ops/maintenance.

Debugging

Scans stuck in queued:

  1. Check GET /api/ops/workers — is there an active, non-stale worker?
  2. If the worker is stale or missing, restart the vega-scan-worker ECS service.
  3. If using SQS, confirm the SQS queue has messages and the worker has permission to receive from it.

Scans stuck in running:

  1. Check whether the ECS runner task is still alive: look for the runner ECS task in the AWS console.
  2. If the task is gone, ReconcileScansUseCase will recover it within VEGA_SCAN_RUNNING_STALE_SECONDS. Reduce this setting to speed up recovery in testing.
  3. Check CloudWatch logs for the runner task for crash indicators.

Quota errors hitting users unexpectedly:

  1. Check GET /api/ops/limits to confirm the active quota values.
  2. Count how many scans the user has in running or queued state.
  3. Cancel old stuck scans to free up capacity.

Billing eligibility errors preventing scans:

  1. Check user billing balance via GET /api/billing/summary.
  2. If using Sub2API, confirm the Sub2API service is reachable from the API container.
  3. Check ScanEligibilityUseCase logs for specific failure reasons.