Podcast
Product Analytics & A/B Testing: Causality, Metrics, Power Analysis, A/A Tests
Open original DataTalks.Club episode
Product Analytics & A/B Testing: Causality, Metrics, Power Analysis, A/A Tests
Original Episode
Use these links for the canonical episode and media sources.
- Open the original DataTalks.Club podcast page
- Watch on YouTube
- Listen on Spotify
- Listen on Apple Podcasts
Episode Overview
How do you design product experiments that truly establish causality and avoid costly false conclusions? In this episode, Jakob Graff — Director of Data Science and Data Analytics at diconium, with prior analytics leadership at Inkitt, Babbel, King and a background in econometrics — walks through practical product analytics and A/B testing strategies focused on causality and reliable metrics.
People
Use these links to connect the episode to guest notes.
Chapter Summary
Use these checkpoints to decide whether to open the source transcript.
- 0:00 - Podcast Introduction
- 1:03 - Guest Background & Career Transition to Data Science
- 5:11 - Econometrics to Product Analytics: Causality Emphasis
- 8:13 - A/B Testing Explained: Clinical Trials Analogy & Randomization
- 11:48 - Experimentation Purpose: Establishing Causality & Controlling Noise
- 14:27 - Case Study: Subscription vs Points — Revenue Metric Design
- 18:06 - De-risking Features & Building Organizational Learning with Experiments
- 23:54 - Experimentation Platform Choices: Third-Party vs In-House
- 24:44 - Traffic Splitter Implementation, Assignment Tracking & Monitoring
- 27:52 - A/A Testing: Validating Randomization and System Trust
- 30:05 - First Test Best Practices: Two-Group Design & Simplicity
- 33:23 - Metric Selection: Noise, Stability, Seasonality & Business Cycles
- 37:44 - Test Duration & Power Analysis: Sample Size Planning
- 40:23 - Statistical Tests Overview: Z-test, T-test, and Nonparametric Options
- 44:39 - Data Distribution Checks: Histograms, Tails, and Visualization
- 47:44 - P-value Intuition: Explaining Significance via A/A Comparison
- 51:55 - Frequentist vs Bayesian Testing: Credible Intervals, Priors & Costs
- 59:08 - Multi-armed Tests (A/B/C/D): Duration, Power, and Multiple Comparisons
- 1:02:52 - Practical Experimentation Tips & Analogies (Pizza Dough)
- 1:03:59 - Hiring, Resources & Contact Information