From ETL to ELT: The Evolution of Data Pipelines in Modern Analytics

Introduction: Why Data Pipelines Evolved

A mid-sized company might operate several production databases, a payments API, a CRM, a marketing platform, a product analytics SDK emitting millions of events a day, and a steady stream of CSV exports from partners. Each system is designed to do its own job well. None is designed to answer a cross-functional question such as: “Which acquisition channel brings in customers with the strongest twelve-month retention?”

Answering that question means bringing data from different sources together in a consistent analytical environment. A data pipeline moves data out of operational systems, prepares it for analysis, and makes it available to analysts, dashboards, and models.

For decades, one common pattern was ETL: extract, transform, load. Data was shaped before it reached the destination, often on dedicated infrastructure. As cloud data platforms matured, ELT became increasingly common: extract data, load it into the analytical platform, and transform it there.

It is tempting to describe this as a newer approach replacing an inferior one. The more useful question is why the preferred architecture changed and when each pattern still makes sense.

What Is ETL?

ETL describes a pipeline built from three sequential stages. Its defining characteristic is the order of operations: data is fully shaped before it reaches its destination, usually on a processing engine that sits between the sources and the warehouse.

Extract

Extraction reads data from the systems that produce it: relational databases such as PostgreSQL or Oracle, REST APIs exposed by SaaS products, flat files dropped onto a file server, application logs, and ERP or CRM platforms. An extract can be a full snapshot of a table or an incremental pull of only the rows that changed since the last run, tracked with an updated-at column or the database's change log. A recurring concern is avoiding load on production systems, which is one reason extraction jobs traditionally ran overnight.

Transform

The extracted data then passes through a transformation engine. Historically this was a dedicated ETL server running tools such as Informatica PowerCenter, IBM DataStage, or SQL Server Integration Services. Typical work includes:

• Cleaning: handling nulls, trimming whitespace, rejecting invalid values.
• Deduplication: removing records that arrived twice or exist in multiple sources.
• Formatting: standardizing dates, currencies, units, and codes.
• Joining: enriching orders with customer attributes from a different system.
• Aggregation: storing daily totals rather than every individual event.
• Business rules: encoding what counts as an “active customer” or how revenue is recognized.

By the end of this stage, the data matches the warehouse schema exactly, typically a dimensional model of fact and dimension tables.

Load

Finally, the transformed data is written into the data warehouse by appending new rows, updating existing ones, or replacing tables outright. Because the shaping already happened upstream, the warehouse receives only clean, conformed, analysis-ready records. Analysts never see the raw inputs.

Why ETL Made Sense

Looking at ETL through a modern lens, it is easy to see inefficiency. Why discard raw data? Why maintain a separate processing tier? But ETL was a rational response to the constraints of its era, and those constraints were real.

Storage was expensive. Warehouses ran on enterprise storage arrays and specialized appliances where every terabyte carried hardware, licensing, and maintenance costs. Loading verbose raw data, including every log line and every unused column, meant paying premium prices for data that might never be queried. Filtering and aggregating first was simply prudent.

Compute was limited and fixed. Warehouse capacity was provisioned up front for peak demand and could not be expanded for an afternoon. The same database served analyst queries and executive reports. Running heavy transformations on it would compete directly with those users, so moving that work to a separate ETL server protected scarce capacity.

Analytical databases were less capable. Many warehouses ran on general-purpose relational engines tuned for well-modeled, pre-aggregated data. Massively parallel platforms such as Teradata did exist, and some teams pushed transformation into the database even then, but that capacity was costly and far from the default.

Control mattered. The warehouse was meant to be a curated source of truth. Transforming upstream acted as a gate: only validated, conformed data got in, and sensitive fields could be kept out entirely.

Volumes were smaller and more predictable. Most data came from a handful of internal transactional systems with stable schemas. Nightly batch windows were long enough to process it, and schemas changed rarely enough that tightly coupling transformation to ingestion was manageable.

Specialized ETL tooling grew up around these conditions, with visual designers, connectors, and scheduling built in. Given the economics, transforming early was the efficient choice. The interesting question is what happened when the economics changed.

The Rise of Cloud Data Warehouses

The change arrived in stages. First, cloud object storage made storing data dramatically cheaper and practically unlimited. Then, in the early-to-mid 2010s, a generation of analytical platforms designed for cloud infrastructure became widely available, including Amazon Redshift, Google BigQuery, and Snowflake. Their individual designs differ, but together they changed the assumptions that ETL was built on which is exactly the shift that made modern data warehousing practical at scale.

• Cheap, scalable storage. Storage became a commodity billed by usage. Keeping raw data stopped being a capital-budget decision and became a modest operating cost.
• Elastic compute. Capacity could be scaled up for a heavy job and released afterward. Serverless models, such as BigQuery's, allocate resources per query, so teams no longer size a cluster for the worst day of the year.
• Distributed, columnar processing. These platforms split data across many nodes that each process a portion in parallel, and they store data by column so queries read only the fields they need, with strong compression. Joins and aggregations over billions of rows, which once required a dedicated processing tier, became ordinary SQL statements.
• Separation of storage and compute. This is arguably the most important architectural change. In a traditional warehouse, storage and compute lived on the same machines, so growing one meant buying both. Snowflake was built around separating them: data sits in cloud object storage while independent compute clusters read from it. BigQuery takes a serverless approach to the same idea, and Redshift adopted it later through managed storage and a serverless option. The practical result is that a large transformation job can run on its own compute without slowing analysts' dashboards, and data can grow without paying for idle processors.

Put together, these changes removed most of the reasons for transforming outside the warehouse. If the destination can store raw data cheaply and run transformations at scale on isolated compute, the case for a separate transformation tier gets much weaker.

What Is ELT?

ELT keeps the same three verbs but reorders them, and the reordering changes where work happens and who owns it.

Extract

As before, data is pulled from databases, APIs, files, and SaaS systems. In practice this is increasingly handled by managed connectors, or by change data capture (CDC), which reads a database's transaction log to replicate inserts, updates, and deletes as they happen.

Load

Data is loaded into the destination in raw or minimally processed form, usually mirroring the source structure and adding metadata such as an ingestion timestamp. It lands in a raw or staging area, sometimes called a landing or bronze layer, that preserves what the source actually sent.

Transform

Transformation runs inside the warehouse, lakehouse, or analytical platform, using that platform's compute and usually expressed in SQL. Raw tables are refined into cleaned staging models, which in turn feed business-level models: facts, dimensions, and metric tables that analysts and BI tools query.

Three differences follow from this ordering. The transformation engine is the analytical platform itself rather than a separate server. Raw data persists alongside the transformed output. And ingestion and transformation become separate concerns that run on their own schedules and are often owned by different people.

Why ELT Became Popular

Once the infrastructure made it practical, ELT offered several advantages that matched how analytics teams actually work.

Let's Build Something Great Together

Ready to transform your idea into a powerful software solution? Talk to our experts and get a free consultation.

Contact Us

• Flexibility. Because raw data is retained, a new question can be answered by writing a new transformation over existing data rather than re-extracting from the source.
• Scalability. Transformation uses the same elastic, distributed engine that serves queries, so it grows with data volume instead of being bounded by a fixed ETL server.
• Faster data availability. Data becomes queryable as soon as it lands. Downstream processing can begin without waiting for an entire transformation chain to finish, although raw data is not automatically trusted data.
• Preservation of raw data. When a bug is discovered in business logic, history can be rebuilt from the raw layer. That matters when sources no longer hold old records, as with many SaaS APIs.
• Easier iteration. Changing a metric definition means editing a SQL model and rebuilding it, not redesigning and redeploying the ingestion pipeline.
• Separation of ingestion and transformation. Ingestion can focus on reliability: schemas, retries, incremental syncs. Transformation can focus on meaning. A failure in one does not automatically break the other.

Technologies That Enabled ELT

Cloud Data Warehouses

Snowflake, BigQuery, and Amazon Redshift are the execution engines for much of the ELT world. Scalable storage made it affordable to keep raw data in the warehouse, and scalable compute made it practical to transform that data where it already sits.

Data Lakes and Lakehouses

Object storage services such as Amazon S3, Azure Data Lake Storage, and Google Cloud Storage can hold files of any format cheaply and at very large scale. A data lake uses this storage to keep raw data, such as JSON, CSV, or Parquet files, without requiring a schema up front. Early lakes struggled with reliability because plain files lack transactions and consistent schemas.

A lakehouse addresses this by adding open table formats such as Delta Lake, Apache Iceberg, or Apache Hudi on top of object storage. These bring ACID transactions, schema enforcement, and versioned tables, so engines like Databricks can run warehouse-style SQL directly over lake data. For ELT, the effect is the same: land raw data, then transform it in place.

Modern Data Ingestion Tools

Fivetran and Airbyte provide prebuilt connectors for hundreds of SaaS applications and databases. They handle authentication, pagination, rate limits, incremental syncs, and schema changes. AWS Database Migration Service (DMS) replicates databases, including ongoing CDC. These tools deliberately do little transformation. Their job is extract and load, which turned much of ingestion from custom engineering into configuration and left modeling to a later stage.

SQL-Based Transformation Tools

dbt became the most visible symbol of ELT, and its role is often misunderstood. dbt does not extract or load data. It lets teams define transformations as SQL select statements, with templating for reuse, and then compiles and runs them inside the warehouse. Its real contribution was bringing software engineering practices to analytics:

• SQL-based transformations that analysts and engineers can both read and write.
• Version control through Git, with code review for data logic.
• Tests that assert properties such as uniqueness, non-null keys, and valid relationships.
• Generated documentation and lineage.
• Modular models that build on one another, with dependencies resolved automatically.

This helped create the analytics engineer role: someone who owns the transformation layer with the discipline of a software engineer.

Cloud-Native Distributed Compute

Underneath a single transformation query, a cloud platform's planner splits the work into many tasks. Each worker scans and processes its own slice of the data, intermediate results are exchanged where joins or groupings require it, and partial results are combined into the final answer. Engines like Apache Spark follow the same principle. From the user's perspective it is one SQL statement; underneath, it may be hundreds of machines working in parallel. That is what makes warehouse-side transformation of very large datasets feasible.

Orchestration and Data Quality

Orchestrators such as Apache Airflow, Dagster, and Prefect do not define ELT, and they orchestrate ETL pipelines just as well. What they provide is operational reliability: scheduling, dependency management, retries, alerting, and backfills. A typical job waits for an ingestion sync to finish, then triggers the transformation run. Data quality checks, from dbt tests to dedicated observability tools that monitor freshness and volume, round out the picture. If you would rather not build and operate this layer in-house, our data orchestration services can help.

ETL vs ELT: What’s Actually Different?

The key difference is the point at which transformation occurs, but that decision affects storage, governance, compute usage, and how quickly teams can change analytical logic.

Neither column is universally better. Each describes a set of trade-offs, and which trade-offs are acceptable depends on the system being built.

Is ELT Always Better?

No. ELT changed the default for cloud analytics, but it did not make ETL obsolete. There are good reasons to transform data before it lands:

• Sensitive data must be filtered before storage. If payment card numbers or health identifiers should never exist in the analytical environment, they must be removed or tokenized in flight.
• Regulations restrict raw-data retention. Data minimization principles, such as those in GDPR, can make “keep everything raw” the wrong default.
• The destination has limited transformation capabilities. Not every target is a scalable warehouse. Operational databases, search indexes, and embedded systems often need data delivered ready to use.
• Data needs substantial preprocessing. Parsing binary formats, processing images, or complex enrichment may be better suited to a general-purpose engine than to SQL.
• Legacy infrastructure exists. Mature ETL pipelines that work reliably are rarely worth rewriting just to follow a trend.
• Specialized requirements apply. Some transformations need custom libraries or performance characteristics that a warehouse does not provide.

In practice, many architectures are hybrids. A pipeline might hash email addresses and drop card numbers during ingestion, load everything else raw, and then do all business modeling inside the warehouse. This pattern is sometimes called EtLT: a light transform in flight, followed by the heavy transformation in the platform.

Modern Data Pipelines: Beyond ETL vs ELT

ETL versus ELT is one design decision inside a much larger system. A modern data platform often combines batch ingestion for SaaS sources with streaming or CDC for databases and event data. It stores data in a warehouse, a lake, or a lakehouse, and frequently a mix of them. It runs transformations that produce curated data models, which feed not just dashboards but machine learning features and data products served back to applications.

Around all of that sit the operational layers: orchestration to coordinate the steps, data quality checks to catch problems before users do, observability to understand what is happening, and governance to control who can see what. A team can make excellent ETL-or-ELT choices and still struggle if these layers are missing.

Choosing Between ETL and ELT

Rather than a rigid decision tree, it helps to think in terms of signals. Each of the following points pushes the design in one direction.

ETL may make sense when:

• Data must be heavily processed before storage.
• Raw data cannot be retained.
• Security or compliance requires processing before data lands.
• The destination has limited processing capabilities.

ELT may make sense when:

• You are using a scalable cloud analytical platform.
• Raw data should be preserved for reprocessing, auditing, or new questions.
• Transformation logic changes frequently.
• Analytics teams need the flexibility to model data themselves.
• Large-scale warehouse-side processing is available.

A hybrid approach may make sense when:

• Some filtering or masking must happen before loading.
• Additional analytical transformations can happen inside the destination.

Most organizations end up with more than one pattern. The goal is to make each choice deliberately, based on the data in front of you.

Conclusion: From ETL to ELT

ETL was the right answer for a world of expensive storage, fixed compute, and smaller, predictable data. ELT became common because that world changed: storage economics shifted, compute became elastic, cloud architecture separated storage from processing, data volumes grew, and analytical requirements began changing faster than pipelines could be redesigned.

The evolution from ETL to ELT is less a story about one pattern defeating another than about infrastructure reshaping where work belongs. The next time that infrastructure shifts, the right place for transformation may move again. The engineers who understand why it moved this time will be ready for it.