Data Engineering & Modernization: Building the Pipelines That Everything Else Depends On

ETL (extract, transform, load) and ELT (extract, load, transform) represent different patterns for moving data through pipelines. In ETL, data is transformed before being loaded into the target system, typically because the target system had limited computational capacity or required specific data structures. In ELT, data is loaded into the target system first and transformed there, taking advantage of modern cloud data platform capabilities for in-database transformation. ELT has become the dominant pattern for cloud data platforms because it is more flexible, allows multiple transformations from the same raw data, and takes advantage of the scalable compute available in cloud environments. Legacy ETL tools are still common but are increasingly being replaced by ELT approaches as organizations modernize their data engineering.

dbt (data build tool) is an open-source framework for transforming data in cloud data warehouses using SQL. It has become popular because it brings software engineering practices to data transformation including version control, testing, documentation, and modular development. dbt allows data teams to develop transformations as code, test them before deployment, and maintain them as collaborative projects rather than as ad-hoc queries. The tool has defined modern patterns for data transformation that have been widely adopted across the industry. Organizations adopting cloud data platforms typically consider dbt or similar tools as part of their transformation approach.

Batch processing collects data over time and processes it in batches at scheduled intervals (hourly, daily, or other frequencies). Streaming processing handles data continuously as it arrives, supporting use cases that need fresh data with minimal latency. Batch processing is simpler to build and operate but cannot support real-time use cases. Streaming processing is more complex but supports use cases including real-time analytics, fraud detection, operational monitoring, and immediate personalization. Most organizations use a mix of batch and streaming based on the specific requirements of each use case. Streaming is not always better than batch, and organizations that implement streaming for use cases that do not actually need it typically create complexity without capturing value.

Legacy ETL modernization requires careful planning because existing pipelines are typically running production workloads that cannot be interrupted. Effective approaches include assessing the portfolio of existing pipelines to identify which should be modernized, which should be replaced, and which should be retired, sequencing modernization work based on business value and technical complexity, running parallel pipelines during transition to validate that new pipelines produce the same results as legacy ones, providing rollback capability when issues emerge during cutover, and building the target architecture incrementally rather than requiring comprehensive completion before any benefits are captured. Organizations that attempt comprehensive modernization without this incremental approach typically encounter problems that force them to abandon or significantly rescope the modernization effort.

Data observability refers to the practices and tools for monitoring the health, quality, and performance of data pipelines and the data they produce. It includes monitoring pipeline execution to detect failures, checking data freshness to ensure data is current, validating data volumes to detect missing data, checking schema consistency to detect structural changes, and monitoring quality metrics to detect degradation. Observability matters because data issues often become visible downstream, when users or applications encounter incorrect data. Effective observability identifies issues at their source before they affect downstream systems, reducing the time and effort required to resolve them. Organizations without observability typically react to data issues after they have caused problems. Organizations with observability identify and address issues before they become visible to users.

Both approaches have their place. In-house data engineering provides deep organizational knowledge, tight integration with business systems, and control over evolution. Managed services provide faster deployment, proven patterns, and reduced operational burden. Many organizations use both, with internal teams handling strategic data engineering and managed services handling commodity integration work. The decision should consider the strategic importance of the specific data engineering work, the availability of appropriate internal expertise, the cost and operational implications of each approach, and the specific vendor options available. Organizations that try to build everything internally often discover that they lack the expertise required. Organizations that try to use managed services for everything often find that their specific requirements do not fit well with standard service offerings.

Data engineering team structure varies based on organizational scale and maturity. Small organizations may have one or a few data engineers embedded in broader data teams. Larger organizations typically have dedicated data engineering functions with specialized roles including platform engineers, pipeline developers, and operations engineers. Very large organizations often have multiple teams with specific responsibilities for different data domains or platform components. Reporting relationships vary, with data engineering sometimes reporting to IT, sometimes to analytics or data functions, and sometimes to specific business units. The right structure should support collaboration between data engineering and the business functions that depend on it, provide appropriate career paths for data engineers, and ensure that data engineering priorities align with broader business priorities.