DataTalks.Club Podcast Wiki

Wiki

Data Engineering

How the DataTalks.Club podcast archive frames data engineering: pipelines, platforms, data quality, role boundaries, business enablement, and the shift toward AI-ready data systems.

Related Wiki Pages

Data Engineering Platforms Data Pipelines Modern Data Stack MLOps and DataOps Data Quality and Observability Analytics Engineering Data Science AI

Data engineering makes data usable and reliable for downstream work. That includes analysis, machine learning, product systems, and business decisions. In the DataTalks.Club archive, data engineers build the paths between source systems and consumers. Those paths cover ingestion, transformation, monitoring, and recovery.

The podcast archive doesn’t treat data engineering as a single fixed job. The early role taxonomy says data engineers prepare product data for analysts and data scientists without burdening operational systems (Data Team Roles Explained). Later interviews split the work across platform engineering, product-facing data engineering, analytics engineering, and DataOps. Recent episodes also add AI-ready infrastructure (Data Engineer Career in 2026, Modern Data Engineering Trends).

This topic is the foundation hub for the broader discipline. For deeper platform patterns, use Data Engineering Platforms. For pipeline operating practices, use DataOps and MLOps and DataOps. For warehouse-side modeling and metric layers, use Analytics Engineering.

Related wiki pages:

Core podcast interviews:

People connected to these discussions:

Common Definition

Across the archive, data engineering converges on one practical definition. Data engineers make data dependable enough for other people and systems to use. They collect data from operational systems and preserve raw context. They also transform data into useful structures, schedule repeatable work, expose interfaces, and monitor whether the work still behaves as expected.

The earliest DataTalks.Club role episode makes the dependency explicit. Data engineers prepare data before analysts query it or data scientists train on it. In that episode, the data engineer also protects the product database from analytical workloads. The same role helps production features receive predictions or prepared datasets when needed (Data Team Roles Explained, about 13:58 and 30:01).

Later episodes add the modern stack vocabulary. Natalie Kwong connects ETL and ELT vocabulary to ingestion and orchestration (ETL vs ELT and Modern Data Engineering, 3:46-15:30 and 30:59-49:32). Jeff Katz’s career episode keeps the foundation more basic. SQL, Python, and data modeling come before Spark for junior engineers (Data Engineering Career Path and Skills, 23:35-38:05).

Disagreements and Boundaries

Guests differ on where the title begins and ends. Slawomir Tulski argues that “data engineer” still hides several jobs. Platform data engineering owns infrastructure, orchestration, access, and shared conventions. Product data engineering sits closer to domain use cases, data products, and stakeholder needs (Data Engineer Career in 2026, 11:54-17:29).

That split explains many archive boundaries. Data engineering overlaps with Data Science around training data, feature pipelines, and production handoff. The role taxonomy episode gives the simple boundary. Data engineers prepare data, while data scientists model and evaluate it. CRISP-DM shows that data collection and preparation can determine whether modeling can even begin (Data Team Roles Explained, 24:55-30:01 and CRISP-DM, 15:46-19:25).

This role boundary is porous because warehouse transformation work often overlaps with analytics engineering.

The clearest references are:

Guests also disagree by architecture pressure. Some scale-up stories need Kafka, schemas, and self-service conventions, but Tulski’s 2026 career discussion warns against treating real-time tools as a maturity badge. Batch or managed systems may fit the business better (Scaling Data Engineering Teams, 12:30-23:26 and Data Engineer Career in 2026, 30:56-38:01).

Pipelines and the Modern Data Stack

Data engineering episodes usually begin with pipeline mechanics because engineers extract source data and transform it for consumers. Kwong’s episode maps this vocabulary. It distinguishes ETL from ELT and places ingestion before dbt-style transformation. It also contrasts lakes with warehouses (ETL vs ELT and Modern Data Engineering, 3:46-12:39 and 30:59-49:32 and Modern Data Stack).

The archive treats those tools as implementation choices. Katz puts SQL, Python, and modeling before distributed systems for beginners (Data Engineering Career Path and Skills, 23:35-38:05). Adrian Brudaru’s trends episode makes the same point from the senior side. Teams should choose platforms and processing tools from actual requirements (Modern Data Engineering Trends, 14:32-18:17 and 35:37-44:42).

Platform Engineering and Self-Service

At team scale, data engineering becomes platform work. Lars Albertsson’s DataOps platform episode covers storage, compute, and workflow engines. The platform lets other teams use data without reinventing the same foundation (DataOps 101 for Scaling Data Platforms, 16:42-35:57 and 50:13-57:46).

Mehdi Ouazza’s scale-up episode adds the human side by showing that self-service needs onboarding and playbooks. Fast-growing teams may need seniors who turn repeated work into shared capabilities (Scaling Data Engineering Teams, 12:30-23:26 and 50:17-54:31 and Self-Service Data Platforms).

This is the main overlap with Data Engineering Platforms and Data Product Management. The platform isn’t successful merely because tables exist. It succeeds when domain teams can safely produce and consume reliable data products. Those products need clear interfaces and ownership (DataOps 101 for Scaling Data Platforms, 57:46-1:04:18 and Last-Mile Data Delivery).

Quality, Observability, and DataOps

The archive repeatedly frames data engineering as reliability work. Pipelines can complete successfully while still delivering late or wrong data. Barr Moses’s observability episode names the key signals. Teams monitor freshness and schema. Lineage and ownership matter too (Data Observability Explained, 16:38-29:00 and 35:24-43:00 and Data Quality and Observability).

Christopher Bergh’s DataOps episodes turn those signals into operating practice. Data engineering teams need tests and CI/CD. They also need observability and recovery behavior. That makes DataOps part of the discipline (DataOps for Data Engineering, 15:52-18:46 and 30:55-54:05 and DataOps).

Quality work also connects data engineering to ML and AI incidents. A model or agent may look broken because an upstream table arrived late. It may also fail because a schema changed or a retrieval corpus lost context. Use MLOps and DataOps when the incident boundary is between model lifecycle and data delivery (DataOps for Data Engineering, 18:46-26:13 and Production-Ready AI Engineering, 18:38).

Batch, Streaming, and Real-Time Tradeoffs

Streaming helps when latency matters, but it isn’t a universal maturity mark.

Use these episodes for the pro-streaming cases:

Use these episodes for the pushback:

Cost, Governance, and Tool Choice

Modern data engineering includes cost and governance. Cloud warehouses and lakehouse stacks can become expensive shared infrastructure. Eddy Zulkifly’s FinOps episode frames data platforms as digital warehouses. Teams need cost tagging, capacity planning, and spend accountability (FinOps for Data Engineers, 21:57-24:34 and 31:40-48:01).

Brudaru’s modern trends episode adds an open-source architecture lens. Iceberg and DuckDB can reduce lock-in, but they still require metadata and governance. The archive’s synthesis is requirements-led tool selection. Choose the smallest system that meets latency and cost constraints (Modern Data Engineering Trends, 18:17-30:31 and 44:42-49:42 and Data Engineer Career in 2026, 25:33-30:56).

AI-Ready Data Engineering

The newer archive links data engineering to AI engineering, but it doesn’t claim that LLMs remove pipeline work. Brudaru describes AI integration as a trend in data engineering and predicts more convergence with AI agents. The same episode keeps the focus on metadata and quality (Modern Data Engineering Trends, 16:40-23:41 and 38:02-44:42).

Production AI discussions show the same dependency because Bartosz Mikulski connects production AI to preprocessing and testing. AI systems still need retrieval corpora and governance (Production-Ready AI Engineering, 18:38 and AI, AI Infrastructure).

Career and Skill Development

The career episodes treat data engineering as an applied engineering path, not a memorized list of tools. Katz recommends Python, SQL, and data modeling before advanced distributed systems. He also presents dbt and Snowflake as early exposure to production data work (Data Engineering Career Path and Skills, 23:35-45:14 and Data Engineering Roadmap).

Tulski adds the market-side distinction. Senior candidates are valued for business judgment, cost awareness, and the ability to avoid over-engineering. He also argues that AI automation makes strategic builders more valuable than people who only operate one narrow tool (Data Engineer Career in 2026, 42:08-1:04:42). Zulkifly’s path from business analysis to data engineering shows why domain understanding and stakeholder translation can become an engineering advantage. That background becomes stronger when paired with cloud, Python, and cost discipline (FinOps for Data Engineers, 6:20-8:18 and 48:01-56:05).

Use these pages for adjacent topics and deeper implementation detail.