XferDB v1.0.0 — First stable release, and bulk-copy for every target adapter
XferDB v1.0.0 — First stable release, and bulk-copy for every target adapter
I just shipped v1.0.0 of XferDB. This is the first release I'm calling stable — every adapter, the full engine (pause/resume/checkpointing), schema analysis, the REST API, the CLI, and the Web UI are complete, and the whole thing has been through a real compatibility matrix run against actual databases, not just go test.
The concrete new thing since v0.5.0: --bulk-copy, XferDB's fast bulk-load write path, now works on every target adapter, not just Postgres.
XferDB's project list — a real Postgres-to-Postgres migration, a SQLite round-trip test, and one intentionally-failed MongoDB run kept around for testing error handling
Download
xferdb (CLI + API server)
| Platform | Binary |
|---|---|
| Linux x86_64 | xferdb-linux-amd64 |
| macOS Apple Silicon | xferdb-darwin-arm64 |
| macOS Intel | xferdb-darwin-amd64 |
xferdb-web (Web UI, optional)
| Platform | Binary |
|---|---|
| Linux x86_64 | xferdb-web-linux-amd64 |
| macOS Apple Silicon | xferdb-web-darwin-arm64 |
| macOS Intel | xferdb-web-darwin-amd64 |
xferdb-web is a separate process — it holds no migration state or database credentials, it just serves the UI and reverse-proxies /api/* to xferdb server. Run it if you want the browser UI; the CLI works standalone without it.
If you'd rather build from source:
git clone https://github.com/bchan77/XferDB.git cd XferDB && git checkout v1.0.0 && go build -o xferdb ./cmd/xferdb
Release notes and CHANGELOG: XferDB v1.0.0 on GitHub.
What's in v1.0.0
Bulk copy for MySQL and SQLite targets
--bulk-copy previously only did anything on a Postgres target (the COPY protocol). It's now implemented for MySQL (LOAD DATA LOCAL INFILE via a registered reader, using REPLACE INTO TABLE so a non-empty target degrades gracefully instead of erroring) and SQLite (batched INSERT OR REPLACE plus PRAGMA synchronous = OFF for the duration of the write). All three still require --truncate or --recreate-schema — bulk copy trades the upsert guarantee for speed, and that tradeoff only makes sense once you already know the target is empty.
It's a real speedup, not a marginal one. At 1M rows per table in our own compatibility runs: mysql267-pg17-bulk finished in 131s versus 742s for the same pair on the default write path — 5.7x. pg16-pg17-bulk was 199s versus 358s — 1.8x.
The migration options screen: schema mode, table/segment workers, and the bulk-copy checkbox — disabled here because "Keep existing target schema/data" is selected instead of truncate or drop & recreate
Adapter test coverage
adapters/postgres, adapters/mysql, and adapters/sqlite had zero unit tests before this release. Building bulk-copy for all three surfaced two real bugs that nothing local would have caught otherwise: Postgres's COPY encoder was hex-mangling non-bytea []byte values (a MySQL-sourced DECIMAL scanned as []byte came out as bytea garbage instead of the number), and SQLite rejects changing PRAGMA synchronous inside an active transaction. Both are fixed, and both now have regression tests.
CI fixes
Two bugs in our own release-gating pipeline, found while shipping this: the Gitea Action that auto-approves a PR once the compatibility matrix passes was sending an invalid API value and silently failing every time — nothing ever checked the HTTP response — and the Jenkins pipeline's flat 240-minute timeout wasn't enough for a full run at L size (1M rows/table). Both fixed.
Showing our work: 45/45 passing
XferDB v0.5.0 shipped with 27/27 passing across three source engines into Postgres. That number is now 45/45: MongoDB 6.0 through 8.3 (21 cases), MySQL 8.0/8.4/9.7/26.7 (17 cases), and Postgres 14 through 17 in every direction (7 cases) — each pairing now includes at least one dedicated bulk-copy case, not just the default write path. Same setup as before: real Postgres/MySQL/MongoDB in Kubernetes, a fresh namespace and databases per case, verified value-for-value on a control table.
Two things I'm not claiming, on purpose:
- SQLite → PostgreSQL is implemented and unit-tested, but SQLite isn't deployable in the matrix the way a client/server database is, so this path has no automated end-to-end coverage yet.
- PostgreSQL → YugabyteDB works — I tested 2025.2.5.2 by hand — but the automated matrix case (pinned to an older 2024.2.11.0-b36) is currently disabled because it hangs during execution. That's a real bug I haven't root-caused yet, not a "should be fine" shrug.
"Tested" on the table in the README means exactly what's in that matrix, automated, on every push. Where it isn't, I'm saying so.
That's the compatibility matrix; here's what a real migration looks like end to end. This is one of my own runs — 1,003,462 rows across 17 tables, including a synthetic 1M-row orders_1m table:
A completed migration: 1,003,462 / 1,003,462 rows (100%), 17 tables, all green
The web UI records a stats snapshot every 5 seconds while a migration runs — rows transferred, rate, phase, table count, CPU and memory — so you're not just watching a spinner:
Stats history: rate climbing from 0 to over 2,700 rows/sec as the migration ramps up, sampled every 5 seconds
Thanks
If you're running MySQL or SQLite as a target and try --bulk-copy for the first time, I'd like to hear how it goes — especially anything outside the versions in the matrix.
— Billy