Skip to content

DBLab Engine

⚡ Blazing-fast PostgreSQL cloning and branching 🐘

🛠️ Build powerful dev/test environments.
🔃 Cover 100% of DB migrations with CI tests.
💡 Quickly verify ChatGPT ideas to get rid of hallucinations.

Available for any PostgreSQL, including self-managed and managed services* like AWS RDS, GCP Cloud SQL, Supabase, and Timescale.

It can be installed and used anywhere: across all cloud environments and on-premises.

Latest release

CI pipeline status Go report

Contributor Covenant Community Slack Twitter Follow


*For managed PostgreSQL cloud services like AWS RDS or Heroku, direct physical connection and PGDATA access aren't possible. In these cases, DBLab should run on a separate VM within the same region. It will routinely auto-refresh its data, effectively acting as a database-as-a-service solution. This setup then offers thin database branching ideal for development and testing.

Why DBLab?

  • Build dev/QA/staging environments using full-scale, production-like databases.
  • Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
  • Automatically test database changes in CI/CD pipelines, minimizing risks of production incidents.
  • Rapidly validate ChatGPT or other LLM concepts, check for hallucinations, and iterate towards effective solutions.

For example, cloning a 1 TiB PostgreSQL database takes just about 10 seconds. On a single machine, you can have dozens of independent clones running simultaneously, supporting extensive development and testing activities without any added hardware costs.

Try it yourself right now:

How it works

Thin cloning is fast because it is based on Copy-on-Write (CoW). DBLab employs two technologies for enabling thin cloning: ZFS (default) and LVM.

Using ZFS, DBLab routinely takes new snapshots of the data directory, managing a collection of them and removing old or unused ones. When requesting a fresh clone, users have the option to select their preferred snapshot.

Read more:

Where to start

Case studies

Features

  • Speed & scale
    • Blazing-fast cloning of PostgreSQL databases – clone in seconds, irrespective of database size
    • Theoretical max of snapshots/clones: 264 (ZFS, default)
    • Maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2128 bytes (ZFS, default)
  • Support & technologies
    • Supported PostgreSQL versions: 10–18
    • Thin cloning (CoW) technologies: ZFS and LVM
    • UI for manual tasks and API & CLI for automation
    • Packaged in Docker containers for all components
  • PostgreSQL containers
    • Popular extensions including contrib modules, pgvector, HypoPG and many others (docs)
    • Customization capabilities for containers (docs)
    • Docker container and PostgreSQL configuration parameters in the DBLab config
  • Source database requirements
    • Location flexibility: self-managed PostgreSQL, AWS RDS, GCP Cloud SQL, Azure, etc.—no source adjustments needed.
    • No ZFS or Docker requirements for source databases
  • Data provisioning & retrieval
    • Simple-mode setup: paste source URL + password, get a probed configuration in one click (UI Configuration → Simple tab)
    • Expert-mode form for both logical (dump/restore) and physical (WAL-G, pgBackRest) retrieval
    • Physical (pg_basebackup, WAL-G, pgBackRest) and logical (dump/restore) provisioning
    • Partial data retrieval in logical mode (specific databases/tables)
    • Database renaming during snapshot creation (databaseRename option)
    • Continuous update in physical mode
    • Periodic full refresh in logical mode without downtime
  • Recovery & management
    • Fast Point in Time Recovery (PITR) for physical mode
    • Auto-deletion of unused clones
    • Time-based auto-deletion of unused branches and snapshots (retention config; safe-only, never force-deletes dependents)
    • Snapshot retention policies in DBLab configuration
  • Clones, branches & snapshots
    • "Deletion protection" for clones, branches, and snapshots (blocks manual and auto-deletion, plus count-based retention for snapshots)
    • Persistent clones withstand DBLab restarts
    • "Reset" command for data version switching
    • "Upgrade" command to move a single clone to a newer PostgreSQL major version in place
    • Resource quotas: CPU, RAM
  • Monitoring & security
    • /healthz API endpoint (no auth), extended /status endpoint (API docs)
    • Prometheus metrics endpoint (/metrics) for monitoring
    • Netdata module for insights

Clone major upgrade

A single clone can be moved to a newer PostgreSQL major version without touching the rest of the instance, which makes it practical to test an upgrade against production-like data:

dblab clone upgrade --target-version 17 my-clone

The same action is available in the UI on the clone page and over the API as POST /clone/{id}/upgrade.

The clone's Postgres is shut down cleanly, pg_upgrade --link converts its data directory in place, and the clone restarts on an image of the target major. Because --link hard-links the data files instead of copying them, the upgrade is fast and consumes almost no extra space. The request returns as soon as the clone enters the UPGRADING state; watch the clone status for the result.

Enabling it. Set provision.pgUpgradeImage to an upgrade image matching the target major. When it is unset the feature is simply unavailable and the endpoint says so; nothing else about the instance changes.

provision:
  pgUpgradeImage: "postgresai/pg-upgrade:17"
  pgUpgradeTimeout: 3h
  pgUpgradePullTimeout: 1h

pgUpgradeTimeout bounds a single upgrade run; it defaults to three hours. When the budget runs out, the upgrade container is removed and the clone is put back on the version its data directory still holds, exactly as it would be after any other failure.

pgUpgradePullTimeout bounds each of the two image pulls that precede it, and defaults to one hour. The upgrade image carries the target major plus the server packages of the four preceding ones, so the first upgrade on an instance downloads several gigabytes; pre-pulling both images makes the step a no-op. Both pulls happen while the clone is still serving traffic, so exceeding the budget leaves the clone untouched and running. The two budgets are separate on purpose — a slow pull must not eat into the time pg_upgrade gets once the clone is already stopped.

An upgrade request may name an image explicitly instead of letting the engine substitute the major in the current tag. Set provision.upgradeImageAllowList to restrict which repositories such a request may name; when it is empty, as it is by default, any repository the instance can reach is accepted.

provision:
  upgradeImageAllowList:
    - "postgresai/extended-postgres"

How it can end. Almost every outcome leaves the clone running:

Status Meaning
OK The clone runs the target version; dbVersion reports it.
WARNING The upgrade was not applied. Either nothing had been converted and the clone still runs its original version, or conversion had begun and the clone was rebuilt from its origin snapshot — in which case data written since the clone was created is lost. The message says which, and points at the pg_upgrade log under <clone dir>/upgrade/logs/.
WARNING, clone not running The upgrade settled on disk but the clone container did not come back up. The data directory is intact and the upgrade is still recorded as pending, so the next engine start finishes or undoes it and brings the clone back.
FATAL Recovery itself failed and there is nothing left to finish. Reset or delete the clone.

Rolling back. Resetting the clone returns it to the version the instance is configured with, because a reset re-provisions from the origin snapshot using the engine-wide image.

Things to know.

  • The target must be newer than the clone's current major and at most four majors ahead — the upgrade image carries binaries for the four preceding versions.
  • Only majors DBLab ships a default configuration for can be targeted (10–18 today).
  • The image tag keeps its extension bundle and glibc suffix; only the major changes (…:16-0.8.0-glibc236…:17-0.8.0-glibc236). Changing the glibc build across an upgrade would change collation behaviour, and pg_upgrade does not reindex. Pass --docker-image to override the choice.
  • pg_upgrade verifies that every extension in the source database has a matching library for the new major. An extension outside the image's set fails this check, and the clone rolls back untouched.
  • Clones using non-default tablespaces are refused: those live outside the clone dataset.
  • In physical mode the sync instance and the pool are untouched — the upgrade only ever acts on a clone's own dataset. The instance-wide dockerImage must still match the source major.
  • Iteration 1 limitation: a snapshot created from an upgraded clone does not record its major version, so clones created from that snapshot start on the instance-wide image and will fail to start. Avoid snapshotting upgraded clones until snapshot-level version resolution lands.

How to contribute

Support us on GitHub/GitLab

The simplest way to show your support is by giving us a star on GitHub or GitLab! ⭐

Add a star

Spread the word

  • Tweet about DBLab and mention @Database_Lab.
  • Share a link to this repository on your favorite social media platform.

Share your experience

If DBLab has been a vital tool for you, tell the world about your journey. Use the logo from the ./assets folder for a visual touch. Whether it's in documents, presentations, applications, or on your website, let everyone know you trust and use DBLab.

HTML snippet for lighter backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>

For darker backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-dark-background.svg" />
</a>

Propose an idea or report a bug

For proposals, bug reports, and participation in development, see our Contributing Guide.

Reference guides

How-to guides

You can find more in the "How-to guides" section of the documentation.

Miscellaneous

License

The DBLab source code is licensed under the OSI-approved open source license Apache 2.0.

Reach out to the PostgresAI team if you want a trial or commercial license that does not contain the GPL clauses: Contact page.

Community & Support

Contributor Covenant

Many thanks to our amazing contributors!

Translations

Making DBLab more accessible to engineers around the globe is a great help for the project. Check details in the translation section of contributing guide.

This README is available in the following translations:

👉 How to make a translation contribution

About

DBLab enables 🖖 database branching and ⚡️ thin cloning for any Postgres database and empowers DB testing in CI/CD. This optimizes database-related costs while improving time-to-market and software quality. Follow to stay updated.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

2.7k stars

Watchers

25 watching

Forks

Releases

Packages

Used by

Contributors

Languages