Skip to content

Source Ingest

Source ingest is the process of getting a repository's code into a form Vega can scan. It now runs as a dedicated asynchronous pipeline with its own worker and runner processes, separate from the scan pipeline.

Intake methods

Git URL

When a user provides a Git URL, CreateRepositoryUseCase records source_kind=git and enqueues an IngestJob. The ExecuteRepositoryIngestUseCase (run by the ingest runner) clones the repository:

  • Shallow clone to avoid downloading unnecessary history
  • Authentication for private repositories (if credentials are configured)
  • Walks the source tree to build a file listing and CLOC metrics

Zip/archive upload

Users can upload a zip or tar archive directly. The route POST /api/repositories/upload-zip handles the multipart upload, then creates a repository record and enqueues ingest. The ingest runner extracts the archive safely using the limits described below.

GitHub App

Repositories connected via the GitHub App (source_kind=github) are fetched by ExecuteRepositoryIngestUseCase using the GitHubPort adapter. The GitHub App installation provides temporary credentials for repository access.

Git upload (programmatic)

GET/POST /api/git-uploads/ creates a temporary git remote that clients can git push to. After pushing, the client calls POST /api/git-uploads/:id/mark-pushed and then POST /api/git-uploads/:id/complete to trigger ingest. This is used for CI-driven workflows.

Ingest pipeline

CreateRepositoryUseCase
    ↓
EnqueueRepositoryIngestUseCase
  → IngestJob(state=queued) written to storage
  → job pushed to SQS ingest queue (or local queue)
    ↓
RunQueuedRepositoryIngestUseCase (vega-repo-ingest-worker)
  → claims IngestJob (state=running)
    ↓
  ├── local mode → ExecuteRepositoryIngestUseCase in-process
  └── ecs mode   → ECSRepositoryIngestRunnerLauncher
                     → vega-repo-ingest-runner ECS task
    ↓
ExecuteRepositoryIngestUseCase
  → clone / extract / GitHub fetch
  → create SourceSnapshot
  → store snapshot to ObjectStoragePort (local dir or S3)
  → transition Repository: creating → snapshotted → ready

IngestJob state lifecycle:

queued → claimed → running → completed
                           → failed
                           → cancelled
                           → stale (reconciled by maintenance)

Stale ingest jobs (stuck in running without a heartbeat) are detected and recovered by ReconcileRepositoryIngestUseCase, called by the ingest worker on each polling loop.

Archive safety

Archive extraction is a security boundary. A malicious user could craft an archive that: - Extracts files outside the intended directory (path traversal, e.g., ../../etc/passwd) - Contains millions of tiny files that exhaust disk space - Has a huge uncompressed-to-compressed ratio (zip bomb)

The ingest runner enforces these limits via adapters/objects/:

Setting Default What it prevents
VEGA_MAX_SOURCE_BYTES 2 GB Archives larger than this are rejected before extraction
VEGA_MAX_ARCHIVE_ENTRIES 50,000 Archives with too many files are rejected
VEGA_MAX_ARCHIVE_UNCOMPRESSED_BYTES 5 GB Total extracted size cap

Path traversal is detected and rejected before any file is written.

Snapshot storage

After ingest, the source is stored as an immutable SourceSnapshot:

Snapshots are stored in directories under data/snapshots/. The SourceSnapshot.storage_uri is a local file path. Scans access source directly from this path.

Snapshots are uploaded as zip archives to the S3 source bucket. The SourceSnapshot.storage_uri is an S3 URI. Scan runner tasks download the snapshot from S3 before scanning. Per-scan S3 credentials are issued via STS (ScanScopedS3Credentials) to limit access scope.

Key settings:

# Local storage (default)
VEGA_FILE_STORAGE_BACKEND=local

# S3 storage (production)
VEGA_FILE_STORAGE_BACKEND=s3
VEGA_S3_SOURCE_BUCKET=vega-prod-source-abc123

SourceSnapshot record

After successful ingest, a SourceSnapshot is created with:

  • snapshot_id — unique ID
  • repository_id — the parent repository
  • commit_sha — the exact commit (for git repositories)
  • tree — file manifest (paths and sizes)
  • sizes — total size metrics
  • cloc — lines of code by language
  • storage_uri — where the archive is stored (local path or S3 URI)

Debugging ingest failures

Repository stuck in creating

  1. Check if the ingest worker is running: GET /api/ops/workers
  2. Check the IngestJob record in storage for the error
  3. Check ingest worker logs for enqueue errors

Repository stuck in snapshotted

The snapshot was created but the runner didn't complete. Check ingest runner logs: 1. In AWS: CloudWatch log group for the ingest runner ECS task 2. Locally: the ingest runner stdout

Git clone failing

  1. Confirm the URL is correct and accessible from the machine running the runner
  2. For private repos, check that git credentials are configured
  3. Check runner logs for specific git error messages

Archive upload failing

  1. Check whether the upload exceeded any of the size limits above
  2. Look for path traversal errors in the logs — archives with ../ paths are rejected
  3. Confirm data/uploads/ is writable (local) or the S3 bucket is accessible (production)

Snapshot upload to S3 failing

  1. Confirm VEGA_FILE_STORAGE_BACKEND=s3 and VEGA_S3_SOURCE_BUCKET is correct
  2. Confirm the ingest runner task role has s3:PutObject permission on the source bucket
  3. Check runner logs for S3 client errors