Datavor v3.5: BigQuery, Redshift, and knowing your sync actually worked

Datavor v3.5 is a reach release. Two new cloud warehouses join the connector lineup, live replication now streams correctly into both, and a new tool tells you — after any sync — whether the target actually matches the source. There's also a small but meaningful story for the agent ecosystem at the bottom.

BigQuery and Redshift join the connectors

Datavor now speaks BigQuery and Redshift natively, alongside the five engines it already supported. That brings the "big three" cloud warehouses under one roof — Snowflake, BigQuery, Redshift — and takes the connector total to seven.

connect_bigquery uses either a service-account key or Application Default Credentials (gcloud auth application-default login — omit the key entirely). One detail worth flagging: BigQuery row counts come from the metadata service, never a billed SELECT COUNT(*) scan. Discovering "how big is this table?" costs nothing.

connect_redshift speaks the PostgreSQL wire protocol and stages bulk writes through S3 COPY. Batches under 100 rows go in as direct INSERTs. Bulk writes and CDC both require an S3 staging bucket and IAM role — pass them at connect time or you'll get a clear error the first time you try to move more than a handful of rows.

Two more warehouses. Same natural-language interface.

Live replication, everywhere it should be

start_cdc now streams correctly into BigQuery and Redshift, not just Snowflake. Each warehouse gets its own writer, matched to how that warehouse actually likes to be written to:

• BigQuery — real MERGE-based upserts, staged via a load job (not the streaming buffer). Nothing lags waiting for the buffer to flush.
• Redshift — temp-table staging plus a transactional upsert (DELETE then INSERT inside a single transaction). No half-applied batches.
• Snowflake — the batch writer from earlier releases, unchanged.

Both new writers carry the source's primary-key metadata through to the target — which is what makes a correct upsert possible on warehouses that don't enforce primary keys of their own. BigQuery has no PK enforcement at all; Redshift's are informational only. Without the source PK carried through, "was this row already here?" becomes ambiguous. Datavor closes that gap for you.

Did the sync actually work?

Every ETL tool eventually collides with this question. You ran the sync, the logs said 45,230 rows, and… now what? The target looks right. Nobody's complaining yet. But between "the pipeline didn't error" and "the target actually matches the source" there's a gap, and it's where data-quality incidents live.

The new verify_sync tool closes that gap. After any sync, it compares source and target on three axes:

• Row counts — always. Free on BigQuery, because it reads metadata rather than scanning.
• Order-independent checksum — a fingerprint that agrees when the two sides carry the same data regardless of row order.
• With method=full — exactly which rows are missing, extra, or duplicated, with sample keys you can go look up.

And because Datavor already remembers everything it's synced, verify_sync also learns. Each table develops a normal row-count range, and anomalies get flagged automatically — "this table is usually 40–50K rows, but now it's 12K." No dashboard to watch. You just get told.

One more thing: reach

Two more sync_table* improvements went in that don't fit neatly under either theme above but matter in practice:

• Different names on each side. sync_table, sync_table_partial, and sync_table_incremental now take an optional target_table. Read a bare MySQL table name, write it into a differently-named or dataset-qualified warehouse table — same call, no renaming step.
• The agent ecosystem. Datavor already worked with any MCP-compatible client — v3.5 packages it for the specific ones agents actually run in. A ready-to-publish OpenClaw skill. Verified Codex CLI compatibility (stdio and Streamable HTTP + bearer-token both confirmed against a real install). And a generic integration guide covering PydanticAI, LangChain / LangGraph, and any other framework that speaks MCP.

The count

v3.5 adds three new MCP tools — connect_bigquery, connect_redshift, verify_sync — bringing the total to 50 tools across 7 databases. It's live in the npm package now.

Install with npx datavor@latest, or just ask your AI: "connect to my BigQuery dataset and verify the customers table matches production." That's the whole setup.