XferDB v1.1.0 — Validation on by default, and proof the copy is right

I just shipped v1.1.0 of XferDB. The headline feature is post-migration validation: after a migration finishes, XferDB now checks that the target actually holds what the source had, table by table, and tells you which rows don't match when it doesn't.

That matters more than it sounds. This release also fixes several bugs in XferDB itself where a migration reported success — row counts and all — while data was quietly missing or wrong. That class of bug is exactly what validation exists to catch, and it's why validation is now on by default.

Download

xferdb (CLI + API server)

Platform	Binary
Linux x86_64	xferdb-linux-amd64
macOS Apple Silicon	xferdb-darwin-arm64
macOS Intel	xferdb-darwin-amd64

xferdb-web (Web UI, optional)

Platform	Binary
Linux x86_64	xferdb-web-linux-amd64
macOS Apple Silicon	xferdb-web-darwin-arm64
macOS Intel	xferdb-web-darwin-amd64

Checksums: SHA256SUMS.

xferdb-web is a separate process — it holds no migration state or database credentials, it just serves the UI and reverse-proxies /api/* to xferdb server. Run it if you want the browser UI; the CLI works standalone without it.

If you'd rather build from source:

git clone https://github.com/bchan77/XferDB.git
cd XferDB && git checkout v1.1.0 && go build -o xferdb ./cmd/xferdb

Full list of changes: CHANGELOG.md.

Before you upgrade: two defaults changed

Validation runs after every completed migration, and xferdb migrate exits non-zero if a table fails it. If you have a script wrapping xferdb migrate, it can start failing where it used to pass — which is the point, but you should know. Turn it off per run with --skip-validation, or per project with transfer.validation: off.
A PostgreSQL source with no Schema set now migrates every non-system schema, not just public. Tables in your connection's own schema keep their bare names, so existing projects and checkpoints carry on unchanged; tables in other schemas come across as schema.table and land in the same schema on a Postgres target. Set Schema: public to get the old behaviour.

What's in v1.1.0

Post-migration validation

Validation re-reads the source the same way the migration did — through the same schema plan, so MongoDB flattening, skipped fields and type overrides reproduce themselves — and compares it with the target. Values are compared by the column's intended type, so true versus 1, 1.50 versus 1.5, JSON key order and timestamp precision don't show up as false differences.

There are three levels:

L1 compares row counts.
L2 (the default) adds an order-independent fingerprint of every row. It catches changed values, not just missing rows, and works on tables with no primary key.
L3 runs on a table that failed L1 or L2 and names the rows: missing from the target, extra in the target, or present with different values (and which columns differ). --sample N limits it to a percentage of a big table.

Two details I'm happy with:

Lossy by plan. If you deliberately narrow a column — numeric(10,4) to numeric(10,2), say — the target rounds every value, and a naive check would flag all of them. The validator applies the same conversion the target does (measured against real PostgreSQL 16 and MySQL, not assumed), and reports a table whose only differences are those as expected lossy rather than failed.
per_table mode validates each table the moment it finishes copying, while the others are still running. On a long multi-table migration you find out about a bad table early instead of at the end.

It's in the CLI (xferdb validate), the API, and the web UI — validation badges on the table list, live progress with an ETA, and a View rows drawer for the L3 differences.

Validation is set per run on the migration form, and the choice is remembered for the project along with the rest of the run options:

The run form's validation options: after the migration or as each table finishes, at L1, L2 or L3, with an L3 sample percentage

Here's a real result: a YugabyteDB to YugabyteDB migration of 1,310,000 rows, with the 1.2M-row orders table checked row by row at L2 against the source:

A completed migration's validation card: all three tables pass, source and target row counts match, and L3 had nothing to diff because L2 already passed

Bugs that reported success while losing data

This is the part I'd want to know about if I were using XferDB:

PostgreSQL paging could skip and repeat rows. By default XferDB reads a table in LIMIT/OFFSET pages, and it sent no ORDER BY. On a large table Postgres doesn't promise a stable order between queries, so pages overlapped. A 2M-row copy wrote exactly 2,000,000 rows — the count looked right — holding only 1,242,222 distinct ids. Pages are now ordered by primary key. If you migrated a large Postgres table with v1.0.0 and didn't use segment workers, it's worth re-running with validation.
Fields missing from a batch's first record were dropped for every row in that batch, on all three target adapters.
Several MongoDB _id shapes (compound, Binary/UUID, floating-point, mixed-type) stopped after the first 1,000 rows with no error; deeply nested documents lost their grandchild fields; and two collections with long names that agreed on their first 63 characters were silently merged into one Postgres table.
MySQL and SQLite targets received source type names unchanged. MySQL rejected Postgres types outright; SQLite accepted them and stored a 20-digit numeric as a float. Both now translate types properly.
SQLite "database is locked" failed tables at random under parallel writes — from XferDB's own state store, not your database.

Everything else

Prometheus metrics at GET /metrics on xferdb server: rows, tables and migrations counters, batch timing histograms, and per-migration progress, rate and ETA gauges.
PostgreSQL schemas — the new default above, plus a Schema setting to narrow to one. Tested against a YugabyteDB source whose tables live outside public.
The web UI lists a server's databases instead of making you type the name, offers to create a missing target database (a button, never automatic, so a typo doesn't quietly become a new database), and remembers your run options per project.

Showing our work: 53/53 passing

v1.0.0 shipped at 45/45. The compatibility matrix is now 53 cases: MongoDB → PostgreSQL (24), MySQL → PostgreSQL (17), PostgreSQL → PostgreSQL (7), and new pairings PostgreSQL → MySQL (2), PostgreSQL → SQLite (2) and MySQL → SQLite (1). Every case now also passes L2 validation, since it runs by default. Same setup as before: real databases in Kubernetes, a fresh namespace per case. Main passed the full set at medium size just before the release was tagged.

What I'm still not claiming:

SQLite → PostgreSQL is implemented and unit-tested, but still has no automated end-to-end coverage.
PostgreSQL → YugabyteDB and YugabyteDB → YugabyteDB are verified by hand on 2025.2.5.2 — including a 1.2M-row table validated at L2 — but the automated YugabyteDB case is still disabled because it hangs on the older version it's pinned to.

Thanks

If validation flags something on one of your migrations, I want to hear about it — whether it's a real difference or a false one. Either way it's a bug worth knowing about.

— Billy