Container registry metadata database

The metadata database provides several enhancements to the container registry that improve performance and add new features.

By default, the container registry uses object storage or a local file system to persist metadata related to container images. This method to store metadata limits how efficiently the data can be accessed, especially data spanning multiple images, such as when listing tags. By using a database to store this data, many new features are possible, including online garbage collection which removes old data automatically with zero downtime.

This database works in conjunction with the storage already used by the registry, but does not replace object storage or a file system. You must continue to maintain a storage solution even after performing a metadata import to the metadata database.

For Helm Charts installations, see Manage the container registry metadata database in the Helm Charts documentation.

Enhancements

The metadata database architecture supports performance improvements, bug fixes, and new features that are not available with legacy metadata storage. These enhancements include:

• Automatic online garbage collection
• Storage usage visibility for repositories, projects, and groups
• Image signing
• Moving and renaming repositories
• Protected tags
• Performance improvements for cleanup policies, enabling successful cleanup of large repositories
• Performance improvements for listing repository tags
• Tracking and displaying tag publish timestamps (see issue 290949)
• Sorting repository tags by additional attributes beyond name

Due to technical constraints of legacy metadata storage, new features are only implemented for the metadata database version. Non-security bug fixes might be limited to the metadata database version.

Known limitations

• Metadata import for existing registries requires a period of read-only time.
• Prior to 18.3, the registry’s regular schema and post-deployment database migrations must be run manually when upgrading versions.
• No guarantee for registry zero downtime during upgrades on multi-node Linux package environments.
• During metadata imports for existing registries, the createdAt and publishedAt timestamp values for image tags are set to the import date. This is intentional to ensure consistency, because the legacy registry does not collect tag published dates for all images. While some images have build dates in their metadata, many do not. For more information, see issue 1384.

Metadata database feature support

You can import metadata from existing registries to the metadata database, and use online garbage collection.

Some database-enabled features are only enabled for GitLab.com and automatic database provisioning for the registry database is not available. Review the feature support table in the feedback issue for the status of features related to the container registry database.

Database requirements

The container registry connects to its metadata database directly and runs read-write transactions against it.

By default, GitLab 18.3 and later preprovisions a logical database for registry metadata in the main GitLab database. This database already meets the requirements below, so most installations need no extra database setup. You meet these requirements yourself only when you use an external database for the registry:

• Run a supported PostgreSQL version. The registry checks the version at startup and does not start on an unsupported version. For supported versions, see PostgreSQL requirements.
• Use a dedicated database for the registry. To create the database and its user, see Configure the container registry metadata database.
• Do not add PostgreSQL extensions. The metadata database needs none, so do not copy the main GitLab database’s extension list.
• Allow read-write access. The registry does not start against a read-only replica.
• Keep bytea_output set to hex, the PostgreSQL default. The registry reads some binary columns as hexadecimal text, so a server or role set to bytea_output = escape makes imports fail with invalid hex format. See Error: convert field 8 failed: invalid hex format.

The registry user must own the database, or migrations fail with a permission error. See Error: permission denied for schema public.

Encrypting the connection with TLS is optional. To encrypt it, set sslmode to require, as in the example configuration. To also verify the server certificate, use verify-full with a CA certificate in sslrootcert.

Enable the metadata database for Linux package installations

Prerequisites:

• GitLab 17.5 is the minimum required version, but GitLab 18.3 or later is recommended due to the added improvements and easier configuration.
• A PostgreSQL database that meets the database requirements. It must be accessible from the registry node.
• If you use an external database, you must first set up the external database connection. For more information, see Using an external database.

Before you start

• After you enable the database, you must continue to use it. The database is now the source of the registry metadata. Disabling it after this point causes the registry to lose visibility on all images written to it while the database was active.
• Offline garbage collection is no longer required. The garbage collection command included with GitLab will safely exit when the database is enabled, but third-party commands, such as the one provided by the upstream registry, will delete data associated with tagged images.
• Verify you have not automated offline garbage collection, especially with a third-party command.
• You can first reduce the storage of your registry to speed up the process.
• Back up your container registry data if possible.
• Configure container registry notifications.
• To display container registry usage statistics on the Usage quotas page, you must configure notifications.

Enable the database for new installations

For installations that have never written data to the container registry, no import is required. You must only enable the database before writing data to the registry.

For more information, see the instructions for new installations.

Enable the database for existing registries

You can import your existing container registry metadata using either a one-step import method or three-step import method. A few factors affect the duration of the import:

• The number of tagged images in your registry.
• The size of your existing registry data.
• The specifications of your PostgreSQL instance.
• The number of registry instances running.
• Network latency between the registry, PostgreSQL, and your configured storage.

You do not need to do the following in preparation before importing:

• Allocate extra object storage or file system space: The import makes no significant writes to this storage.
• Run offline garbage collection: While not harmful, offline garbage collection does not shorten the import enough to recoup the time spent running this command.

Choose the right import method

If you regularly run offline garbage collection, use the one-step import method. This method should take a similar amount of time and is a simpler operation compared to the three-step import method.

If your registry is too large to regularly run offline garbage collection, use the three-step import method to minimize the amount of read-only time significantly.

If you use an external database, make sure you set up the external database connection before proceeding with a migration path.

For more information, see Using an external database.

Restore interrupted imports

Skip repositories that you pre-imported within the last 72 hours to resume interrupted imports. Repositories are pre-imported either:

• By completing step one of the three-step import process
• By completing the one-step import process

To restore interrupted imports, configure the --pre-import-skip-recent flag. Defaults to 72 hours.

For example:

# Skip repositories imported within 6 hours from the start of the import command
--pre-import-skip-recent 6h

# Disable skipping behavior
--pre-import-skip-recent 0

For more information about valid duration units, see Go duration strings.

Post import

After completing a large import, hundreds of thousands or even millions of blobs can be queued for garbage collection review. This is normal.

Because tagged images are imported before dangling blobs are inventoried, the garbage collector initially reviews blobs that are still referenced by tagged images. Garbage collection removes these blobs from the queue, but does not delete them from storage.

Storage decreases only after the garbage collector reaches the dangling blobs. Registry storage might take 48 hours or more to decrease after a post-import, because the garbage collector delays review to avoid interference with image blobs.

To monitor and manage the post-import garbage collection backlog:

• Check the health of online garbage collection to see the size and status of the review queues.
• Adjust the garbage collector worker interval to temporarily speed up processing for large backlogs.

Prefer mode

Prefer mode is a configuration option for the metadata database that lets the registry fall back to legacy metadata storage when an existing registry has not been imported to the database yet.

Enable prefer mode

To enable prefer mode:

1. In /etc/gitlab/gitlab.rb, set database.enabled to 'prefer' instead of true or false:

   ruby

   registry['database'] = {
     'enabled' => 'prefer',
     'host' => '<your_database_host>',
     'port' => 5432,
     'user' => '<your_database_user>',
     'password' => '<your_database_password>',
     'dbname' => '<your_database_name>',
   }

2. Save the file and reconfigure GitLab.

After you reconfigure GitLab, the registry evaluates which metadata backend to use at startup based on lockfiles that track previous writes to the filesystem or database.

Prefer mode lockfile workflow

The registry writes lockfiles at startup and the metadata import writes one when it completes, but a push never writes one.

A registry using legacy metadata writes filesystem-in-use when it starts against storage that already contains metadata. A registry using the metadata database writes database-in-use at startup, regardless of what storage holds. The metadata import also writes database-in-use when it completes.

The following table shows the outcome for each lockfile state under enabled: false and enabled: 'prefer'. It assumes storage holds a complete catalog and the configured database is reachable, holds the registry schema, and contains no repository rows.

The last row of the table is intended for a site that has not yet served a push from the database. On a site that has, the registry starts and serves a catalog missing everything pushed after the import ran. For more information, see Revert to object storage metadata.

Only the third row fails without reporting an error. The others either serve the complete catalog or refuse to start and name the reason.

A missing lockfile does not always mean the storage is empty. Any registry that has not restarted after its storage was first populated has no lockfile. This includes a Geo secondary site filled through its registry API by replication. It also includes storage restored from a snapshot or mirrored from another site, unless the copy included a lockfile.

If the database is not usable (for example, the server accepts connections but rejects the configured credentials), the registry does not start and gitlab-ctl reconfigure also fails at the registry database migration. For more information on how to avoid both the empty catalog and the failure to start, see Existing installations.

Emptying or recreating storage does not clear the lockfiles. The lockfiles directory is a sibling of v2/ rather than inside it. Deleting v2/blobs and v2/repositories to reclaim space leaves filesystem-in-use in place. The registry then falls back to legacy metadata again at the next start against storage that is now empty. In that state, pushes succeed and tags appear, while the metadata database stays empty, which can read as an unfinished import. If you intend to move to the metadata database and nothing was pushed after the storage was emptied, remove the lockfile. Stop the registry, delete filesystem-in-use from the directory described in Check lockfiles on disk, and start the registry. If anything was pushed after the storage was emptied, run the metadata import instead, because removing the lockfile leaves those images out of the catalog. If you want to stay on legacy metadata, keep the lockfile.

The fallback decision occurs once at startup and does not change while the registry is running. There is no automatic retry or reconnection to the database after a fallback. To move from filesystem to database mode after a fallback, complete the standard metadata import and restart the registry.

Default configuration

In GitLab 19.0 and later, the metadata database is enabled by default in prefer mode for new installations:

• Linux package (Omnibus) installations: registry['database']['enabled'] defaults to 'prefer' when the setting is not specified in /etc/gitlab/gitlab.rb. For more information, see issue 9396.
• Self-compiled installations: database.enabled defaults to 'prefer' when the setting is not specified in the registry configuration file.

After upgrading, check which backend the registry uses. For the procedure, see Verify which metadata backend is active.

New installations

On a new GitLab 19.0 or later installation, the registry starts in prefer mode. If a reachable metadata database is configured, the registry uses it. Without a reachable database, the registry fails to start.

To keep a new installation on filesystem metadata, set the database mode to false before the first registry start:

• For Linux package (Omnibus) installations, in /etc/gitlab/gitlab.rb:

  ruby

  registry['database']['enabled'] = false

• For self-compiled installations, in /home/git/gitlab/config/gitlab.yml:

  yaml

  registry:
    database:
      enabled: false

Existing installations

Upgrading an existing installation to GitLab 19.0 or later preserves the current registry['database']['enabled'] setting. The upgrade does not migrate metadata or switch the active backend.

An existing prefer-mode installation with filesystem metadata continues to use filesystem metadata after the upgrade. To switch to the database, complete a metadata import.

If registry['database']['enabled'] is not set in /etc/gitlab/gitlab.rb, the upgrade leaves the setting unspecified, and the registry starts in prefer mode. Before upgrading such a site, confirm the lockfile is present. For the procedure, see check lockfiles on disk.

If the registry uses legacy metadata and filesystem-in-use is absent, restart the registry once before you upgrade:

sudo gitlab-ctl restart registry

The restart writes the lockfile, so prefer mode falls back to legacy metadata correctly. Without the restart, the registry finds no lockfile and takes the no-lockfile path described in Prefer mode.

Metadata database backups

When the registry uses the metadata database, include the registry database in your backups. For the procedure, see Backup with metadata database.

A prefer-mode installation with existing filesystem metadata stays in fallback across restarts. During fallback, the registry does not read from or write to the metadata database. You do not need to back up the metadata database until fallback ends.

To end fallback, complete a metadata import and restart the registry. After the restart, the registry uses the metadata database. Include it in your backup routine.

Verify which metadata backend is active

To verify which metadata backend your registry is using, use one of the following methods.

Check the registry API response header

The registry requires authentication on the /v2/ endpoint. The endpoint answers an unauthenticated request with 401 Unauthorized and none of the gitlab-container-registry-* headers, so the check produces no output rather than an answer.

To check the registry API response header:

1. Get a token for the registry. The registry accepts only a JWT issued by GitLab. Request one from the registry token endpoint. Authenticate with a personal access token that has at least the read_registry scope. You do not need to be an administrator. This request does not need a scope parameter, because /v2/ requires a valid token rather than access to a repository:

   shell

   token=$(curl --silent --user "<username>:<personal-access-token>" \
     "https://gitlab.example.com/jwt/auth?service=container_registry" \
     | grep --only-matching '"token":"[^"]*"' | cut --delimiter='"' --fields=4)

   If the personal access token has no registry scope, the endpoint returns a 403 Forbidden status code and an empty token.

2. Send a request to the registry /v2/ endpoint:

   shell

   curl --silent --head --header "Authorization: Bearer ${token}" "https://registry.example.com/v2/" | grep --ignore-case gitlab-container-registry-database-enabled

3. Inspect the gitlab-container-registry-database-enabled response header:

   • A value of true means the registry is using the metadata database.

   • A value of false means it is using legacy filesystem storage.

   • No output means the header was absent, so the command answered nothing rather than false. An absent header does not mean the metadata database is disabled. Confirm the response status before you act on the result:

     shell

     curl --silent --output /dev/null --write-out '%{http_code}\n' --header "Authorization: Bearer ${token}" "https://registry.example.com/v2/"

     A 401 means the request was not authenticated, either because token is empty or because the token was rejected. If you cannot authenticate, use Check lockfiles on disk or Check registry logs instead.

Check lockfiles on disk

To check lockfiles on disk, look for these files in the configured storage backend at <rootdirectory>/docker/registry/lockfiles/:

• database-in-use: The registry is using the metadata database.
• filesystem-in-use: The registry is using legacy filesystem storage.

If both lockfiles exist, the registry is in an invalid state and does not start.

Check registry logs

The registry logs which metadata backend it selects at startup.

To check registry logs, look for one of the following messages:

• If the registry falls back to legacy storage (prefer mode only):