Skip to content

Overlay-base database size cap (7,500 MB) disables incremental PR analysis for large JavaScript/TypeScript repos #4327

Backlog

Description

@hernan82arg

Summary

Incremental (overlay) analysis never kicks in for our pull requests, because the default branch's
javascript-typescript overlay-base database is larger than the hard-coded upload cap in
src/overlay/caching.ts (OVERLAY_BASE_DATABASE_MAX_UPLOAD_SIZE_MB = 7500). Every default-branch
run logs:

Database size (11832 MB) exceeds maximum upload size (7500 MB). Skip uploading overlay-base database to cache.

so every PR run finds no overlay-base database and falls back to a full analysis.

Environment

  • codeql-action v4.39.3, CodeQL CLI 2.27.4, code scanning default setup on self-hosted runners
    (16 vCPU / 56 GiB), languages: actions, c-cpp, javascript-typescript, python.
  • A large pnpm monorepo: about 22,000 JS/TS files extracted after paths-ignore (tests, specs,
    stories, e2e, tooling) is already applied through a config file.

Numbers

  • Overlay-base database on the default branch: about 11.7 GB uncompressed, measured by the action.
  • The same file set built locally as a plain database (codeql database create, no overlay mode):
    about 1.6 GB. So the overlay-base form is roughly 7x the plain database for us.
  • PR analysis time: about 17 minutes for javascript-typescript on 16 cores, all of it a full
    re-analysis. Before the repo grew past the cap, PR runs took 2 to 7 minutes.
  • Trimming more paths cannot get us under the cap without excluding production code.

Ask

One of:

  1. Make the cap configurable (an input or environment variable) for repos whose cache budget allows it.
  2. Apply the cap to the compressed size the cache actually stores (the code comment estimates
    about 3 GB compressed at 2.5:1).
  3. Reduce the overlay-base database size before upload, if some of it is not needed to restore the
    base for PR analysis.

Happy to share more measurements if useful.

Activity

  1. jketema commented on Nov 16, 2026

    @jketema

    Hi @hernan82arg,

    We'll have a look to see whether we can make the cap configurable.

  2. jketema commented on Nov 18, 2026

    @jketema

    Small update: the 7x size difference is unexpected, although it is somewhat on the higher side of what we see (average is something like a 4x size difference).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions