Podcast
ML System Design Interviews: Production ML, Fraud Detection, Features, A/B Testing & MLOps
Open original DataTalks.Club episode
ML System Design Interviews: Production ML, Fraud Detection, Features, A/B Testing & MLOps
Original Episode
Use these links for the canonical episode and media sources.
- Open the original DataTalks.Club podcast page
- Watch on YouTube
- Listen on Spotify
- Listen on Apple Podcasts
Episode Overview
How do you approach ML system design interviews that probe production constraints, fraud detection trade-offs, and MLOps realities? In this episode, Valerii Babushkin — Senior Director of Data, Analytics, and AI at BP, Kaggle Competitions Grandmaster, and author of Machine Learning System Design — walks through what interviewers look for and how candidates should structure answers for real-world ML problems.
People
Use these links to connect the episode to guest notes.
Chapter Summary
Use these checkpoints to decide whether to open the source transcript.
- 0:00 - Podcast Introduction & Episode Overview
- 1:51 - Valerii Background: Career Snapshot and Kaggle Achievements
- 3:21 - Blockchain.com Role: Scope, Responsibilities, and Data Ownership
- 5:46 - Transition to Meta: User Privacy Work and Large-Scale ML Experience
- 7:31 - Hiring Experience: Conducting High-Volume Interviews and Team Leadership
- 9:12 - Candidate Targeting: Who Faces ML System Design Interviews
- 11:23 - Interview Structure: 45-Minute Narrative and Evaluation Goals
- 13:58 - Contrast: Software System Design Versus ML System Design
- 13:58 - Fraud Detection Case Study: Probabilities, Loss Functions, and Real-Time
- 16:43 - Labeling, Class Imbalance, and Feature Engineering Tradeoffs
- 20:33 - Interview Tactics: Stating Assumptions and Getting Alignment
- 22:05 - Example: Points-of-Interest System vs Personalized Recommender
- 24:28 - End-to-End ML Pipeline: Metrics, Baselines, and A/B Testing
- 29:09 - Securing the Interview: Iterative Baselines and Signposting Depth
- 31:58 - Appropriate Depth: Practical ML Decisions vs Research-Level Detail
- 33:31 - Preparation Strategies: Mock Interviews, Resources, and Experience
- 37:59 - Industry Checklist: Core ML Project Review Items and Patterns
- 40:11 - Defining Goals and Proxy Metrics: Business Alignment and Long-Term Health
- 44:11 - Features, Labels, Model Selection, and Validation Workflow
- 46:02 - Production Robustness: Monitoring, Distribution Shift, and Fallbacks
- 47:52 - System Components: Why Features Matter More Than Model Architecture
- 50:57 - Engineering Integration: Serving Models, Embeddings, and MLOps Roles
- 52:25 - When to Avoid ML and Useful Design Pattern References
- 54:07 - New Grad Expectations: Coding Focus and Limited System Design
- 57:23 - Validating in Production: A/B Tests, Causality, and Human Labels
- 59:01 - Career Path: Moving from Data Science Practice to System Design