Podcast
Scale Data Engineering Teams: Build Self-Service Data Platforms, Hire Senior Engineers & Use Kafka
Open original DataTalks.Club episode
Scale Data Engineering Teams: Build Self-Service Data Platforms, Hire Senior Engineers & Use Kafka
Original Episode
Use these links for the canonical episode and media sources.
- Open the original DataTalks.Club podcast page
- Watch on YouTube
- Listen on Spotify
- Listen on Apple Podcasts
Episode Overview
How do you scale data engineering teams during hypergrowth without sacrificing quality or developer velocity? In this episode, Mehdi OUAZZA — a data engineer and entrepreneur with 7+ years working on streaming and batch pipelines, data modeling, orchestration, infrastructure and analytics — walks through practical approaches to scale data engineering teams, build self-service data platforms, hire senior engineers and adopt Kafka-based event streaming.
People
Use these links to connect the episode to guest notes.
Chapter Summary
Use these checkpoints to decide whether to open the source transcript.
- 1:57 - Episode Introduction: Growing Data Engineering Team & Guest Mehdi
- 2:42 - Guest background: BI, on-prem Big Data to staff data engineer (career highlights)
- 5:41 - Defining scale-up: hypergrowth, funding, hiring surge, speed vs quality
- 10:21 - Hypergrowth challenges: product launches, US expansion, operational strain
- 12:30 - Data platform role: enabling self-service, onboarding, and scalability
- 17:22 - Data platform anatomy: Airflow, conventions, playbooks, and best practices
- 20:13 - Hiring for scale: prioritize senior experts and niche technology experience
- 23:26 - Event streaming practices: Kafka, schemas, schema registry, and data contracts
- 27:05 - Velocity vs growth: managing fast pace while ensuring personal growth
- 31:07 - Culture shift: evolving processes and influencing company norms
- 35:05 - Career trade-offs: scale-up vs enterprise vs FAANG
- 38:12 - Assessment tactics: reverse interviews to evaluate team workload and culture
- 39:02 - Junior opportunities: rapid learning, promotions, and exposure in scale-ups
- 40:51 - Talent sourcing: employer brand, community contributions, and open source
- 46:44 - Technical content: writing, OSS contributions, and getting external feedback
- 49:06 - Community engagement: reader outreach, calls, and mentorship benefits
- 50:17 - Role evolution: generalist to specialist as teams and projects mature
- 52:55 - Work balance: platform engineering vs use-case pipelines (~50/50)
- 54:31 - Path to senior: proactivity, broader impact, and cross-team collaboration
- 56:34 - Casual segment: light banter about music, caps, and hobbies
- 57:48 - Creator spotlight: MehdiO DataTV, DataCreators.club, and content channels
- 1:00:12 - Content production: time investment, process improvements, and persistence
- 1:01:53 - Video editing tips: multi-take filming, lighting consistency, and tricks