Skip to content

Data Model

This page describes the core entities in Vega and how they relate to each other. All entities are Pydantic DomainModel instances stored as JSONB documents in Postgres (or JSON files locally) — there is no ORM and no per-entity SQL schema beyond the payload jsonb column.

Entity hierarchy

erDiagram
    WORKSPACE_PROJECT {
        string project_id
        string user_id
        string name
        string description
        timestamp created_at
        timestamp updated_at
    }
    REPOSITORY {
        string repository_id
        string user_id
        string parent_project_id
        string source_kind
        string state
        string snapshot_id
        string default_scope
        string special_threat_modeling_id
        timestamp created_at
    }
    SOURCE_SNAPSHOT {
        string snapshot_id
        string repository_id
        string commit_sha
        json tree
        json sizes
        json cloc
        string storage_uri
        timestamp created_at
    }
    INGEST_JOB {
        string job_id
        string repository_id
        string state
        int attempt
        string worker_id
        string runner_task_arn
        timestamp heartbeat_at
        timestamp completed_at
    }
    SCAN_RECORD {
        string scan_id
        string repository_id
        string snapshot_id
        string state
        json options
        list include_paths
        list bug_classes
        string depth
        bool fast
        json shards
        json billing_fields
        json llm_usage_metrics
        timestamp created_at
        timestamp completed_at
    }
    FINDING_RECORD {
        string finding_id
        string scan_id
        string repository_id
        string title
        string status
        string severity
        string confidence
        json buggy_locations
        json verification_fields
        json triage_metadata
        timestamp created_at
        timestamp updated_at
    }
    ARTIFACT_RECORD {
        string artifact_id
        string scan_id
        string artifact_kind
        json object_ref
        string access
        string stage
        string component
        string shard
        timestamp created_at
    }
    DOMAIN_EVENT {
        string aggregate_id PK
        int sequence PK
        string event_type
        json payload
        timestamp occurred_at
    }
    RUNNER_JOB {
        string job_id
        string scan_id
        string ingest_job_id
        string phase
        string state
        string ecs_task_arn
        timestamp lease_at
        timestamp heartbeat_at
    }

    WORKSPACE_PROJECT ||--o{ REPOSITORY : "contains"
    REPOSITORY ||--o{ SOURCE_SNAPSHOT : "has"
    REPOSITORY ||--o{ INGEST_JOB : "tracked by"
    REPOSITORY ||--o{ SCAN_RECORD : "has"
    SOURCE_SNAPSHOT ||--o{ SCAN_RECORD : "scanned by"
    SCAN_RECORD ||--o{ FINDING_RECORD : "produces"
    SCAN_RECORD ||--o{ ARTIFACT_RECORD : "produces"
    SCAN_RECORD ||--o{ DOMAIN_EVENT : "emits"
    SCAN_RECORD ||--o{ RUNNER_JOB : "tracked by"

Entity descriptions

WorkspaceProject

A project is the top-level workspace container. Users create projects to group related repositories and security work together.

Key fields: project_id, user_id, name, description, timestamps.

In the code: domain/projects/records.py (Pydantic model), stored via ProjectStore port.

Repository

A repository is the source code target inside a project. It can be added in three ways:

  • Git URL — the ingest pipeline clones the repository
  • Zip upload — the user uploads an archive; the pipeline extracts it
  • GitHub App — the GitHub integration fetches the repository

After a repository is added, it goes through an ingest pipeline. The state field tracks this:

creating → snapshotted → ready

Additional transient states during scanning: scanning, completed.

Key fields: repository_id, user_id, parent_project_id, source_kind (git / zip / github), GitHub metadata fields, state, ingest progress, snapshot_id, default_scope, special_threat_modeling_id.

In the code: domain/repositories/records.py, stored via RepositoryStore port.

SourceSnapshot

A snapshot is an immutable, point-in-time capture of a repository's source code. When the ingest runner completes, it stores a snapshot with the file tree, size metrics, and a reference to the stored archive.

Scans always run against a specific snapshot. This means you can run multiple scans against the same snapshot, and results are always tied to a known state of the code.

Key fields: snapshot_id, repository_id, commit_sha, tree (file manifest), sizes, cloc, storage_uri.

In the code: domain/ingest/snapshots.py, stored via SnapshotStore port.

IngestJob

An ingest job tracks the lifecycle of a single repository ingest operation. It is created when a repository is added and transitions through states as the ingest pipeline runs.

State lifecycle:

queued → claimed → running → completed
                           → failed
                           → cancelled
                           → stale (detected by maintenance)

Key fields: job_id, repository_id, state, attempt, worker_id, runner_task_arn, heartbeat and completion timestamps.

In the code: domain/ingest/records.py, stored via IngestJobStore port.

ScanRecord

A scan is one audit run against a repository snapshot. It has a rich lifecycle with optional pausing.

State lifecycle:

queued → running → completed
                 → failed
                 → cancelled
                 → paused → running (resumed)

A scan optionally includes include_paths to scope the audit to specific directories, bug_classes to restrict vulnerability categories, and fast mode for a single-pass audit.

A scan optionally runs in fast scan mode (controlled by ScanCapacityPolicy.fast_scan_enabled()): plan → threat_model → audit all execute inline in one ECS task, avoiding round-trips. In standard mode, separate ECS runner tasks are launched per phase.

The shards field is an embedded list of dicts inside the ScanRecord JSONB payload — not separate database records. Each shard dict tracks one unit of work:

Shard phase What runs Key fields in shard dict
plan Planning phase shard_id, state, runner_task_arn
threat_model Threat model generation same
audit Per-component audit + component_ids, plan_artifact_ref, threat_model_artifact_ref
verify Finding verification + finding_ids, verification_runner_task_arn

Verification shards are grouped by ScanCapacityPolicy (max findings per runner, max concurrent runners per scan), and are populated after audit completes via _queue_verification_work().

Key fields: scan_id, repository_id, snapshot_id, state, options, include_paths, bug_classes, depth, fast, PR context, shards (embedded list), billing fields, LLM usage metrics.

In the code: domain/scans/records.py, domain/scans/states.py (state machine), stored via ScanStore port.

FindingRecord

A finding is a specific security issue the scan engine discovered. Unlike events (which are an append-only log), findings are structured records intended for human review and triage.

Each finding includes:

  • severity — critical, high, medium, low, or info
  • title — brief description of the issue
  • status (triage) — candidate, triaging, confirmed, dismissed, needs_more_evidence
  • confidence — scan engine's confidence level
  • buggy_locations — file paths and line ranges where the issue was found
  • verification_fields — verification status and timing if verification was run

Findings are upserted (created or updated) as the scan engine emits finding_updated events. If the engine reports the same finding twice, the backend updates the existing record rather than creating a duplicate.

In the code: domain/findings/records.py, application/findings/ use cases, stored via FindingStore port.

ArtifactRecord

An artifact record references a file produced by the scan (stored in S3). Artifacts have an access level (operator or customer) and are categorized by artifact_kind:

  • vega_core_events — raw vega-core event stream (vega-core-events.jsonl)
  • debug_bundle — full debug bundle zip
  • generated_threat_model — the threat model Markdown file
  • activity_log — structured activity log for the scan
  • worker_components_log — per-component worker status log

Key fields: artifact_id, scan_id, artifact_kind, object_ref (URI + SHA256 + size), access, stage, component, shard.

In the code: domain/artifacts/records.py, application/artifacts/ use cases, stored via ArtifactStore port.

DomainEvent

Domain events are the append-only record of everything that happened during a scan. They are written by the scan engine adapter and read by the frontend for the live scan feed.

Key fields: aggregate_id (typically scan_id), sequence (ordering per aggregate), event_type, payload, occurred_at.

Events come from two sources:

  • Backend-emitted (by ExecuteScanUseCase): scan_running, scan_completed, scan_failed, planning_artifact_reused, scan_runner_started. Each backend event is published under both aggregate_id=scan_id and aggregate_id=repository_id.
  • Engine-emitted (by vega-core via EngineEventSink, stored under aggregate_id=scan_id): scan_started, scan_progress, scan_log, finding_updated, component_worker_*, stage_started, stage_completed, scan_completed, scan_cancelled, finding_verified.

In the code: domain/events/records.py, stored in the domain_events table via EventStore port. The PK is (aggregate_id, sequence).

RunnerJob

A runner job tracks the ECS task that executed a scan phase or ingest. It stores the ECS task ARN so the worker can monitor task health, detect failures, and cancel running tasks.

Key fields: job_id, scan_id or ingest_job_id, phase, state, ecs_task_arn, lease and heartbeat timestamps.

In the code: domain/operations/runner_jobs.py, stored via RunnerJobStore port.

Generic records (via GenericRecordStore)

Many auxiliary entities use the same JSONB document pattern without dedicated store interfaces. These include:

Record type Purpose
users User profiles
api_keys Programmatic API key credentials
refresh_tokens Auth refresh token lifecycle
sessions Legacy upload/analyze sessions
git_uploads Temporary git-push remote metadata
github_connections GitHub App installation records
worker_heartbeats Worker process health tracking
planning_artifacts Cached plan bundles (avoid re-running planning)
special_threat_modeling Per-repository custom threat modeling config
billing_* Promotions, entitlements, spend limits
sub2api_* Per-scan LLM API key lifecycle
usage_ledger Billing debit/credit ledger
finding_daily_rollups Pre-computed finding counts by day
workspace_revisions Audit trail for workspace mutations

Persistence

All domain models extend DomainModel (Pydantic v2, extra="ignore"). Storage is schemaless JSONB keyed by record_key:

-- Example: the scans table
CREATE TABLE scans (
    record_key   TEXT PRIMARY KEY,
    payload      JSONB NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

The application stores the full Pydantic model as payload. This means:

  • No migration needed for new fields — new fields are simply added to the Pydantic model with a default
  • Queries use JSONB operators — e.g., payload->>'state' = 'running' or payload @> '{"state":"running"}'
  • Migration files handle schema evolution when tables, indexes, or structural changes are needed

In local development, all entities are stored as JSON files under data/. This makes it easy to inspect and reset state without a database.

In production, entities are stored in Postgres via AWS RDS. The schema is defined in SQL migration files:

app/storage/migrations/
├── 001–011  ← legacy migrations (initial schema through planning artifacts)
├── 012–021  ← refactored architecture migrations
├── 022      ← domain_events table with sequence ordering
├── 023–027  ← billing, ingest, GitHub, scan key, artifact tables
└── 028      ← scan log read models (segments + worker components)

See Migrations for the full list and details.

Large objects (source code archives, scan reports, debug bundles) are always stored in S3, not Postgres. The Postgres payload holds the S3 object reference.