Scan Lifecycle

A scan is the central operation of Vega. This page traces the complete journey from the moment a user clicks "Start scan" to when findings appear in the dashboard. The exact path differs between local mode and AWS production mode.

Complete scan flow

flowchart TD
    A["User clicks 'Start scan' in dashboard"]
    B["POST /api/repositories/:id/scans\n(or /api/scans)"]
    C["CreateScanUseCase\n→ eligibility check (billing)\n→ write ScanRecord status=queued\n→ enqueue scan job"]

    A --> B --> C

    C --> D{Execution mode?}

    D -->|thread| E["API runs ExecuteScanUseCase\nin a background thread"]
    D -->|external| F["Scan worker polling loop\nclaims the scan"]
    D -->|sqs| G["API sends message to SQS scan queue"]
    G --> F

    F --> H["RunQueuedScanUseCase\n→ claim ScanRecord (status=running)\n→ ExecuteScanUseCase"]
    H --> I{scan_worker_execution_mode?}
    I -->|local| J["ExecuteScanUseCase runs\nvega-core in-process via\nLocalVegaCoreService"]
    I -->|ecs| K["ECSScanRunnerLauncher\ncalls ECS RunTask\nlaunches vega-scan-runner container"]

    J --> L
    K --> L

    L["VegaCoreEngineAdapter\n→ run_plan() / run_audit() / etc."]
    L --> PA{Planning artifact\navailable?}
    PA -->|yes| M2["Reuse cached planning artifact\n(skip planning phase)"]
    PA -->|no| M["vega-core planning stage\nCodex → component list + threat model"]
    M --> M3["Store planning artifact bundle\nUpload threat_model.md to S3"]
    M2 --> N
    M3 --> N
    N["vega-core audit stage\nCodex analyzes each component\n(parallel thread pool)"]

    N --> P["Engine emits DomainEvents:\nfinding_updated, scan_progress,\ncomponent_worker_*, etc."]
    P --> Q["EngineEventSink maps events\nto domain types"]
    Q --> R["NormalizeAndUpsertFindingsUseCase\nupserts FindingRecord per finding_updated event"]
    Q --> S["EventStore appends DomainEvent\n(sequence-ordered)"]
    R --> VQ{Verification\nenabled?}
    VQ -->|yes| VR["Queue verification shard\nFindingRecord.verification_status=queued"]
    VR --> VS["Worker launches ECS verify runner\nrun_verification() on vega-core"]
    VS --> VE["FindingRecord.verification_status\n→ running → completed / failed"]
    VQ -->|no| T
    R --> T

    T["RecordArtifactsUseCase\nuploads output to S3\nArtifactRecord created"]
    T --> U["ScanRecord state → completed\n(or failed / cancelled)"]

    U --> V["Dashboard polls /api/scans/:id/live\nand /api/findings/scans/:id"]

Step-by-step explanation

1. Scan is created

The user initiates a scan from the dashboard or via API. CreateScanUseCase:

1. Checks billing eligibility via ScanEligibilityUseCase
2. Creates a ScanRecord in the database with state=queued
3. Enqueues the scan job (to SQS, a local queue, or directly starts a thread depending on scan_execution_mode)
4. Returns a 202 Accepted response immediately — it does not wait for the scan to finish

Optional scan parameters include include_paths (scope to specific directories), bug_classes (restrict vulnerability categories), and fast mode.

2. Worker claims the scan

RunQueuedScanUseCase (called by the scan worker or triggered by thread mode):

1. Reads the next queued scan from the queue
2. Atomically transitions the ScanRecord from queued → running (claiming prevents double-execution)
3. Calls ExecuteScanUseCase with the claimed scan

3. Runner is launched (ECS mode)

If scan_worker_execution_mode=ecs, ECSScanRunnerLauncher calls AWS ECS RunTask to start a vega-scan-runner container. The scan ID is passed as an environment variable. The runner builds its own RuntimeContainer via build_scan_runner_runtime() and calls ExecuteScanUseCase.

In local or thread mode, ExecuteScanUseCase runs directly in the worker or API process using LocalVegaCoreService.

4. Source is loaded

ExecuteScanUseCase resolves the SourceSnapshot for the scan. In production, this means reading the snapshot metadata from Postgres and using the storage_uri to download the archive from S3. Locally, the snapshot is read from a directory under data/.

5. Planning (with caching)

VegaCoreEngineAdapter.run_plan() invokes the vega-core planning stage. The engine uses Codex to analyze the codebase structure and produces:

• A component manifest (which files/modules to audit)
• A generated threat model (threat_model.md)

If a matching planning artifact already exists for the same repository, snapshot, and include-paths, PlanningArtifactUseCase retrieves the cached bundle and planning is skipped entirely. This saves significant LLM cost on repeat scans.

When planning completes, the bundle is stored via RecordArtifactsUseCase for the audit phase to use.

6. Per-component auditing

There are two audit paths depending on scan configuration:

Standard (ECS sharded) mode — run_audit() is called per-shard with a batch of component_ids assigned to that runner. Multiple runners audit different components in parallel.

Fast scan mode — When both ScanCapacityPolicy.fast_enabled == True (policy level) AND scan["fast"] == True (per-scan option) are set, and no pre-assigned component IDs are present, ExecuteScanUseCase._run_fast_scan() runs plan → threat_model → audit fully inline in one process. Plan discovers components, threat model runs, then ScanCapacityPolicy.component_groups() splits them into batches of max_workers_per_runner (capped to max_fast_runners_per_scan shards). Each audit shard runs sequentially. All artifact refs are threaded between stages. This avoids ECS task round-trips.

In both modes, VegaCoreEngineAdapter.run_audit(): - Receives the plan artifact ref and component IDs - Calls the vega-core audit stage, which runs Codex in a thread pool - For each component, Codex reads source files and produces structured findings - All Codex calls route through the LLM proxy (via OPENAI_BASE_URL), which enforces per-scan usage limits

7. Events and findings stream back

Events come from two sources and are carefully distinguished:

Backend-emitted events — ExecuteScanUseCase._publish_scan_event() emits lifecycle events directly. Each backend event is published twice: once to aggregate_id=scan_id and once to aggregate_id=repository_id (with scan_id added to payload). This lets both the scan-level and repository-level event streams stay in sync.

Backend event type	When
scan_running	Scan transitions to running state
scan_completed	Scan finished (all phases)
scan_failed	Scan failed with an exception
planning_artifact_reused	Cached planning artifact was used (skipping plan)
scan_runner_started	ECS runner task was launched for a verification shard

Engine-emitted events — vega-core calls EngineEventSink during execution:

Engine event type	Meaning
scan_started	Engine audit loop has begun
scan_progress	Progress update (stage name, component N of M)
scan_log	Debug log line from vega-core or Codex
finding_updated	A security issue was discovered; payload contains BugRecord list
component_worker_started	A component audit worker has started
component_worker_completed	A component audit worker finished
component_worker_failed	A component audit worker failed
scan_completed (engine)	vega-core finished the audit loop
scan_failed (engine)	vega-core encountered an error
scan_cancelled (engine)	Scan was cancelled cooperatively
finding_verified	Verification result for one finding
stage_started / stage_completed	Stage-level progress (used for plan cache replay)

All events (both backend and engine) are stored as DomainEvents in the EventStore. For finding_updated engine events: NormalizeAndUpsertFindingsUseCase additionally creates or updates FindingRecords.

8. Finding verification (optional)

When a finding is upserted, the backend optionally queues it for verification. Verification is controlled by max_verification_runners_per_scan in ScanCapacityPolicy (default 0, disabled). When enabled:

1. Each new FindingRecord gets verification_status=queued
2. The scan worker's reconcile loop detects queued verification shards and launches ECS verify-phase runner tasks
3. Each verify runner calls VegaCoreEngineAdapter.run_verification() — a separate vega-core invocation running only the verify stage
4. Finding verification_status transitions: queued → running → completed (or failed)
5. The frontend displays the verification status on each finding

9. Artifacts are written

RecordArtifactsUseCase and MaterializeScanLogArtifactsUseCase write scan outputs:

• vega-core-events.jsonl — all raw vega-core events
• vega-core-debug-bundle.zip — full debug bundle (for diagnosing scan failures)
• threat_model.md — generated threat model from the planning stage
• activity-log — structured scan activity log (operator-readable)
• worker-components-log — per-component worker status log

10. Dashboard updates

The frontend polls GET /api/scans/:scan_id/live for scan status and GET /api/findings/scans/:scan_id for findings. Once the scan is completed, all findings and artifacts are available for review.

---

Execution mode quick reference

Local (simplest):
  VEGA_SCAN_EXECUTION_MODE=thread
  → API process runs ExecuteScanUseCase directly in a background thread

Local (production-like):
  VEGA_SCAN_EXECUTION_MODE=external
  VEGA_SCAN_WORKER_EXECUTION_MODE=local
  → Separate worker process, in-process vega-core, no ECS

Production:
  VEGA_SCAN_EXECUTION_MODE=sqs
  VEGA_SCAN_WORKER_EXECUTION_MODE=ecs
  → SQS queue, ECS worker, isolated ECS runner per scan phase

---

Scan phases and shards

In ECS mode, the backend distributes work across multiple shard tasks for parallelism. Shard records are stored in the shards field of the ScanRecord:

Phase	What runs	Key env var
plan	Planning + threat model (one shard per scan)	—
threat_model	Threat model generation (if separated)	—
audit	Per-component audit	VEGA_ASSIGNED_COMPONENT_IDS
verify	Finding verification	VEGA_ASSIGNED_FINDING_IDS

Each shard tracks its phase, state (pending / running / completed / failed), ECS task ARN, and the assigned component or finding IDs.

---

Planning artifact caching

Planning artifacts cache the output of the planning phase to avoid re-running planning for subsequent scans against the same snapshot. An artifact includes:

• The component manifest (module-level plan files)
• The generated threat model (threat_model.md)
• A fingerprint of the include-paths and planner version

The artifact is keyed by repository, snapshot, include-paths hash, and planner hash. Artifacts expire and are cleaned up by the maintenance job.

---

Scan control operations

Users can control a running scan:

Action	Use case	What happens
Cancel	CancelScanUseCase	Sets a cancellation flag; if ECS runner is running, ECS StopTask is called; partial findings are preserved
Pause	PauseScanUseCase	Marks the scan paused; the runner cooperatively checks and stops between stages
Resume	ResumeScanUseCase	Re-queues the paused scan from its current phase
Retry	RetryScanUseCase	Re-queues a failed scan from the beginning

---

Stale scan recovery

A scan that stays running longer than VEGA_SCAN_RUNNING_STALE_SECONDS (default: 6 hours) is treated as stale. ReconcileScansUseCase (called by the scan worker on each loop iteration) detects stale scans and marks them failed with an appropriate error event. This handles cases where the runner task crashed silently without writing a failure event.