The story of enterprise data infrastructure in the 2010s is largely a story of failed data lakes. The premise was compelling: store everything in object storage at low cost, schema-on-read instead of schema-on-write, no upfront modeling required. The reality was data swamps. Thousands of files with no consistent naming, no schema documentation, no lineage, no quality controls. Data that the data science team could not use because they could not trust it. Data engineering backlogs filled with requests to clean up messes that should never have been created.
The reaction in many organizations was to retreat to the data warehouse: structured schemas, enforced data types, reliable query performance, governed data products. But warehouses have their own constraints. They are expensive at the storage and compute scales that ML workloads require. They handle unstructured data poorly. Their rigid schemas make iterative data science work slow and frustrating. They are designed for SQL analytics, not for training neural networks.
The data lakehouse architecture addresses both failure modes: it brings warehouse-grade reliability and governance to lake-scale storage, while maintaining the flexibility that ML workloads need.
What a Lakehouse Solves
The core innovation of the lakehouse architecture is ACID transaction support on top of object storage. ACID (Atomicity, Consistency, Isolation, Durability) transactions are what make relational databases reliable: a write either happens completely or not at all, concurrent reads see a consistent state, and committed data is not lost. Object storage (S3, GCS, Azure Data Lake Storage) does not natively provide ACID guarantees. Open table formats (Delta Lake, Apache Iceberg, Apache Hudi) add this capability on top of object storage, making the storage layer behave like a database in terms of reliability while retaining the cost and scale characteristics of object storage.
Schema enforcement becomes optional rather than mandatory. You can enforce a strict schema on tables that serve downstream BI consumers who need reliability. You can allow flexible schemas on tables that serve exploratory data science workloads where the schema is still evolving. The governance layer enforces the right level of strictness for each use case rather than applying warehouse-style rigidity everywhere.
Both BI and ML workloads can run against the same storage layer. The SQL analytics team uses Trino or Databricks SQL to run structured queries for their dashboards. The ML team reads the same Parquet files into Spark or Ray for feature engineering and model training. The data is stored once, governed once, and served to multiple compute engines.
The Storage Layer
Object storage (AWS S3, Google Cloud Storage, Azure Data Lake Storage Gen2) is the foundation. The cost-per-terabyte of object storage is dramatically lower than block storage or warehouse storage. For organizations with petabyte-scale data, this difference is material. The practical consequence is that it is economically viable to retain raw data indefinitely in object storage, rather than having to make hard decisions about what to keep and what to discard.
Parquet has become the de facto standard format for lakehouse storage because it is columnar (efficient for analytical queries that read a subset of columns), compressible, widely supported across compute engines, and self-describing. Raw data typically arrives in other formats (JSON events, CSV exports, Avro from Kafka) and is transformed to Parquet as part of ingestion.
Open table format selection in 2026: Delta Lake is the most mature and most widely adopted, with the deepest integration into the Databricks ecosystem. Apache Iceberg has stronger multi-engine support and is the better choice when you need to read the same table from multiple compute engines (Spark, Flink, Trino, Athena) without engine-specific dependencies. Apache Hudi is optimized for high-frequency upserts and is well-suited to streaming ingestion use cases where records are frequently updated. Most greenfield implementations in 2026 choose Iceberg for maximum flexibility or Delta Lake for Databricks-heavy shops.
The Compute Layer
Apache Spark remains the dominant compute engine for large-scale data transformation and ML preprocessing. Databricks, which provides a managed Spark environment with additional tooling, has become the dominant platform for enterprise lakehouse implementations. Trino (and its commercial distributions) is the go-to for interactive, low-latency SQL queries where Spark's batch-oriented architecture would add unnecessary overhead. DuckDB is emerging as a powerful option for single-node analytical workloads that fit in memory or near-memory scale, with impressive performance for use cases that do not require distributed compute.
Serverless vs cluster-based compute is primarily a cost optimization decision. Serverless compute (Athena, Databricks Serverless, BigQuery) scales to zero when idle and charges per query, which is economical for intermittent workloads. Cluster-based compute has higher fixed costs when running but lower per-query costs for continuous workloads. Most enterprise lakehouses use both: serverless for ad-hoc exploration, clusters for scheduled production pipelines.
For organizations building or modernizing their data and AI platforms, the compute layer selection should be driven by the workload mix (batch heavy vs real-time heavy, SQL heavy vs Python heavy) rather than by vendor preference alone.
The Catalog Layer
The catalog is the directory of your data: what tables exist, where they are stored, what their schemas are, who owns them, and who can access them. Without a catalog, the lakehouse is just files in object storage with no way to discover or govern them at scale.
Unity Catalog (Databricks) is the most capable enterprise catalog available in 2026 for Databricks-based lakehouses, providing unified governance across tables, files, ML models, and dashboards in a single control plane. AWS Glue Data Catalog is the native option for AWS-heavy organizations. Apache Hive Metastore is the legacy standard that most open-source tools still support as a baseline. Organizations not committed to a single vendor ecosystem often choose Apache Polaris or OpenCatalog for REST catalog compatibility across multiple compute engines.
The Governance Layer
Data lineage tracks where data came from and what transformations it went through to reach its current state. For regulated industries, lineage is a compliance requirement: auditors want to know the provenance of numbers in financial reports. For ML teams, lineage is an operational requirement: when a model's performance degrades, the first question is whether the training data changed. Open Lineage is the emerging standard protocol for lineage collection across different compute engines.
Access control in a lakehouse needs to operate at multiple levels: which users can see which databases and tables, which columns within a table (for PII protection), and which rows (for multi-tenant data isolation). Column-level and row-level security are now table-stakes requirements for any enterprise lakehouse serving regulated data.
PII handling requires identifying which tables and columns contain personal information, applying appropriate masking or tokenization for non-privileged users, and ensuring that raw PII is not replicated into development or test environments without appropriate controls. This is a governance policy implemented in the catalog and enforced at the compute layer.
AI and ML Readiness
A standard lakehouse supports BI and analytics. An AI-ready lakehouse adds three additional components. A feature store manages the engineering, storage, and serving of ML features: the transformed and aggregated representations of raw data that models are trained on. Feature stores prevent the common problem of features being recomputed differently by different teams for different models. A model registry tracks which models exist, what data they were trained on, what their performance metrics are, and which version is deployed in production. An embedding store provides vector storage and similarity search for retrieval-augmented generation (RAG) applications and semantic search use cases that are now standard components of enterprise AI systems.
For a detailed treatment of how these components fit together in an AI-native data platform, the companion post on data platform modernization guide covers the modernization path from existing systems to a lakehouse architecture. For a look specifically at the fintech and banking sector, the specific compliance, real-time, and risk data requirements shape the lakehouse architecture in ways that are worth addressing separately.
Real Architecture Decisions
Partitioning strategy is one of the most consequential early decisions in a lakehouse design. Partition columns should match the most common query filter patterns. Time-based partitioning (by date or hour) is appropriate for event data and log data. Entity-based partitioning (by region, product, customer segment) is appropriate for transactional data. Wrong partitioning choices produce query plans that scan far more data than necessary, making queries slow and expensive.
Ingestion patterns: batch ingestion is simpler to implement, appropriate for data sources that update on a schedule (daily ERP exports, nightly database dumps). Streaming ingestion using Kafka and Flink is required when freshness requirements are under an hour, for event-driven data sources, or when downstream consumers need near-real-time data access.
The serving layer for low-latency queries is a separate concern from the lakehouse storage. The lakehouse is optimized for analytical throughput, not millisecond response times. Applications that need sub-second query responses (dashboards with high-frequency refresh, customer-facing applications, real-time operational reports) typically use a materialized serving layer: aggregated results computed in advance and cached in a fast-access store (DynamoDB, Redis, Postgres, BigQuery BI Engine).
Migration Path from Existing Systems
The lift-and-shift approach to lakehouse migration, moving all existing data warehouse tables to the lakehouse simultaneously, consistently produces worse outcomes than the phased approach. The phased approach identifies the two or three use cases where the lakehouse provides the most value over the existing system (typically: ML workloads that the warehouse cannot serve, large raw data volumes that are too expensive to warehouse, new streaming data sources that the warehouse cannot ingest), builds the lakehouse for those use cases first, and expands from there as each phase proves out.
Running the warehouse and the lakehouse in parallel during migration is normal and expected. The goal is not to decommission the warehouse on day one of the lakehouse. The goal is to migrate workloads to the lakehouse as each migrated workload delivers better results than its warehouse equivalent.
