Lakehouse without the lock-in

The lakehouse promised the best of both worlds: the flexibility and cost profile of a data lake with the reliability and semantics of a warehouse. The catch is that the easiest path to one runs straight through a single vendor's formats, catalog, compute, and pricing. You get the architecture, and a lock-in bill to go with it.

Here is the part that changed, and that most architecture decks haven't caught up with: in 2026 the file format is no longer where lock-in lives. That war is effectively over. Lock-in moved up the stack to the catalog and down the stack to egress pricing - where moving a single petabyte across clouds still costs $50,000 to $90,000 in transfer fees alone. This post walks through where portability actually lives now, with the concrete tests we use when we design platforms for clients - so that the day you want to switch engines, the switching cost is a catalog connection change, not a two-quarter migration.

Table of contents
1. What the lakehouse actually promised
2. The format war is settled - build on it
3. The catalog is the new lock-in
4. Egress: the quiet exit fee
5. A portability test you can run this quarter
6. The reference boundaries that keep you portable
7. What openness costs you - the honest tradeoffs
8. When lock-in is a fine trade
9. Takeaways

1. What the lakehouse actually promised

Strip the marketing and the lakehouse is three commitments:

• One copy of the data, in open files on cheap object storage, instead of a lake for data science and a separate warehouse copy for BI.
• Warehouse semantics on that copy: ACID transactions, schema evolution, time travel - the things that made warehouses trustworthy.
• Engine choice: Spark for pipelines, Trino or DuckDB for interactive queries, a warehouse engine for BI, your ML stack for training - all reading the same tables.

The third commitment is the one vendors quietly renegotiate. The first two are now table stakes; every major platform delivers them. Whether you keep engine choice depends entirely on two decisions people still treat as implementation details: which catalog owns your tables, and where the bytes physically sit.

2. The format war is settled - build on it

For years, "which open table format" was the agonizing choice: Apache Iceberg, Delta Lake, or Hudi. That debate has resolved enough that you should stop spending meeting time on it.

Databricks now supports Iceberg as a first-class format - notable because Databricks created Delta Lake. AWS made Iceberg the native option with S3 Tables. Snowflake has committed to full bidirectional Iceberg interoperability. Iceberg v3 closed the last meaningful feature gaps (deletion vectors, row lineage, a VARIANT type) and reached general availability on Snowflake in May 2026.

The practical guidance falls out cleanly:

• Default to Iceberg for a new multi-engine lakehouse. Its specification-driven design is exactly what keeps tables readable when you change engines, clouds, or vendors.
• Delta is the right answer inside its strongholds. If you live in Databricks or Microsoft Fabric, Delta plus UniForm (which writes Iceberg-compatible metadata alongside) gives you the ecosystem's best performance without closing the exit.
• Raw capability has converged. With Iceberg v3 and Delta 4.0, the feature matrix is mostly a wash - the deciding factor is your platform, not the format.

Either way, your data ends up as Parquet files plus a metadata layer, sitting in object storage you control. That is the foundation of every portability argument that follows. If your platform's "lakehouse" stores data in a proprietary internal format with an export path, you do not have a lakehouse; you have a warehouse with a lake-shaped brochure.

3. The catalog is the new lock-in

Here is the failure mode we now see most often in architecture reviews: every table is pristine open-format Parquet, and none of it is reachable, because the catalog that owns the table metadata only lets blessed engines write - sometimes only blessed engines read.

The catalog decides three things: who can discover tables, who can read them, and who can commit to them. Even with a fully open file format, a closed catalog re-creates the lock-in at a different layer. You cannot point a new engine at the data because the engine cannot connect to the catalog, so teams end up fragmenting - a second catalog for the new engine, duplicated governance, and eventually duplicated data. That is the old two-copy world back again, with extra steps.

What good looks like in 2026 is convergence on the Iceberg REST Catalog API as the neutral interface:

• Databricks Unity Catalog exposes managed Iceberg tables over the Iceberg REST API, and can govern Iceberg tables that live in external catalogs (Glue, Hive Metastore, Snowflake Horizon).
• Snowflake's Horizon Catalog, as of March 2026, allows external engines to write to Snowflake-managed Iceberg tables in public preview - the pattern is no longer "Snowflake writes, others read" but "one catalog governs, any engine reads or writes."
• Apache Polaris and other open catalogs give you the same interface without a platform attached.

The test to apply is one sentence long: can a third-party engine you do not pay the vendor for connect to the catalog and both read and write a table? If the answer involves the word "roadmap," you have found your lock-in.

4. Egress: the quiet exit fee

Storage stays portable right up until the network bill makes moving it expensive. Cloud egress pricing is the least discussed and most durable lock-in mechanism in the stack, because it is not a product decision you made - it is a property of where your bytes sleep.

Do the arithmetic once and it stays with you. At typical list egress rates of roughly $0.05 to $0.09 per GB, moving a petabyte across clouds costs on the order of $50,000 to $90,000 in transfer fees alone, before you count the weeks of pipeline work, dual-running costs, and validation. For a proprietary warehouse, add the export step: every table serialized out to Parquet or CSV first, at petabyte scale a weeks-long job in itself. This is why "we can always migrate later" is usually a statement about intent rather than economics.

Three developments are bending the curve, and belong in your evaluation matrix:

• Zero-egress object storage. Cloudflare R2 charges nothing for data egress, and its R2 Data Catalog is a managed Iceberg catalog built into the bucket. For workloads where compute location varies, storing the canonical copy where reads are free changes the calculus.
• Cross-cloud query layers. Google's cross-cloud lakehouse work queries Iceberg tables sitting in S3 or Azure without copying bytes, using caching to keep hot data local. The source cloud still charges egress on cold reads, but the cost curve flattens dramatically.
• Interconnect pricing. If you are genuinely multi-cloud, private interconnect plus VPC endpoints is not an optimization, it is the difference between a viable architecture and a punitive one.

The design rule we apply: put the canonical copy of the data where leaving is cheapest, and bring compute to it - not the other way around.

5. A portability test you can run this quarter

Portability claims are cheap. Here is the drill we run with clients, usually in under two weeks, that converts the claim into evidence:

• Week 1: stand up a foreign engine. Pick one production-grade table. Connect an engine your primary vendor does not sell - Trino and DuckDB are the usual candidates because they speak Iceberg natively and cost nothing to trial - through the catalog's REST endpoint. No data copies allowed.

-- The whole test, from the foreign engine's side (Trino):
-- if this works against your catalog without an export job,
-- your lakehouse is as open as the slide deck says.

SELECT count(*), max(updated_at)
FROM iceberg.prod_catalog.orders
WHERE order_date >= DATE '2026-06-01';

INSERT INTO iceberg.prod_catalog.orders_smoke_test
SELECT * FROM iceberg.prod_catalog.orders LIMIT 100;

• Week 2: measure the exit. Take your largest table. Compute the real cost and wall-clock time to relocate it: egress rate times bytes, plus the metadata rewrite if your catalog does not travel. Write the number down and put it in the platform documentation. A migration cost you have measured is a negotiating position; one you have not is a hostage situation.

Two outcomes are common. Either the test passes and you have durable proof your options are open - worth real money the next time your vendor's renewal quote arrives - or it fails somewhere specific (catalog auth, write path, a UniForm gap), and now you have an engineering ticket instead of a vague unease.

6. The reference boundaries that keep you portable

Lock-in resistance is not a product you buy; it is four boundaries you refuse to blur:

• Storage you own. Buckets in your cloud account, with your keys and your lifecycle policies. Vendor-managed storage that you access through the vendor is a different product with the same name.
• An open table format at rest. Iceberg by default, Delta with UniForm inside its strongholds, per section 2.
• A catalog that speaks the Iceberg REST API to strangers. Managed is fine - Unity, Horizon, Glue, Polaris all qualify today - as long as the foreign-engine test in section 5 passes.
• Compute treated as a commodity. Engines attach to the catalog; nothing engine-specific leaks into the data layer. Materializations, table maintenance (compaction, snapshot expiry), and access policy live with the catalog and format, not inside one engine's proprietary features.

Notice what is absent: nothing above says "avoid managed platforms." You can run this entire architecture on Databricks or Snowflake and be genuinely portable. The boundaries are about which features you adopt and where the metadata lives, not about self-hosting for its own sake.

7. What openness costs you - the honest tradeoffs

Anyone who tells you the open path is free is selling something too. The real costs, so you can budget for them:

• You inherit table maintenance. Iceberg tables need compaction, snapshot expiration, and orphan-file cleanup. Managed platforms do this invisibly; on the open path someone owns the maintenance jobs and their failure modes. It is not hard, but it is not zero.
• The best features land inside walls first. Vendor-native tables get the newest optimizations (liquid clustering, automatic statistics, serverless everything) months or years before the open equivalents. You are trading the frontier for the exit.
• Integration surface is yours to test. "Every engine reads Iceberg" is true the way "every browser renders HTML" is true. Version skew between engine, format spec, and catalog implementations is real; budget integration testing time, especially at spec transitions like v2 to v3.
• Governance is only as portable as its enforcement. Fine-grained policies defined in one catalog do not automatically travel. If you use vendor-specific row- and column-level security, document it as a known migration cost.

Our experience: for a platform team of reasonable seniority, the openness overhead is a low-single-digit percentage of platform engineering time. Against a migration that would otherwise cost quarters, it prices like insurance - but price it, don't hand-wave it.

8. When lock-in is a fine trade

Portability is a means, not a religion. Cases where we advise clients to take the vendor-native path with eyes open:

• Small data, small team. Under a few terabytes with two data engineers, migration is a weekend, not a quarter. The exit is already cheap; optimize for velocity instead.
• One ecosystem, deliberately. If the organization has standardized on one platform end to end and the switching scenario is genuinely implausible inside five years, vendor-native features are simply better value. Write the decision down with its expiry date.
• The feature is the business. Occasionally a proprietary capability is the product edge. Take it - and isolate it, so the blast radius of that dependency stays one workload wide, not platform wide.

The anti-pattern is not choosing lock-in; it is choosing it by default, unpriced, one convenient feature at a time, and discovering the accumulated position at renewal.

9. Takeaways

• The format war is over; default to Iceberg, take Delta inside Databricks/Fabric, and stop debating it.
• Lock-in now lives in the catalog and in egress. Evaluate both explicitly.
• Apply the one-sentence catalog test: a foreign engine must read and write through the catalog's REST API.
• Measure your exit cost once a year, in dollars and weeks. A measured migration is leverage; an unmeasured one is a hostage note.
• Openness costs low-single-digit engineering overhead - real, budgetable, and usually worth it as insurance.
• Lock-in can be a rational trade. Make it consciously, in writing, with an expiry date.

References

1. Databricks: Announcing full Apache Iceberg support
2. Snowflake: Open by Design - commitment to Iceberg and interoperability
3. Databricks: What is an open lakehouse?
4. Lakehouse table formats in 2026: Iceberg, Delta, Hudi, Paimon, DuckLake
5. ClickHouse: Open table formats and the path to no-lock-in observability
6. Google Cloud: Build a cross-cloud open data lakehouse without copying bytes

Hero image: Taylor Vick on Unsplash.

We design data platforms that fit your stack today and still fit it after it changes. Explore our other insights or get in touch if you would like to talk it through.