Hi Denis Lepchev thanks!
You bring up a really good point about ‘feedback loops’ when dealing with continued model evaluation. There is a lot written about this. I remember reading an article about a company called Stripe that discussed this problem for them in some detail for a credit fraud model they had developed.
We discuss this briefly in the subsection called “Capturing ground truth” as well as other considerations. You mention a good approach. Another technique to consider is to use causal impact models. Causal impact uses Bayesian techniques to understand how a certain “treatment” might have affected a population. It’s commonly used in scenarios where it’s not possible (or not ethical) to have a hold out or control group. This is commonly used at Google and that team has open sourced a CausalImpact package you might want to check out.

Thank you for the answer, Mike.

Thanks so much Alper Demirel. I’m glad you enjoy it. 🙂

• In my mind, this book would come after an intro book or reference on ML. We certainly talk about how you would do things (and there is a github repo with code implementing the ideas in code) but this book is focused more on common problems and their solutions that we’ve learned in working with customers along the years. Likely a lot of patterns will be familiar to you, if you’re a practicing ML/AI engineer. So this book is a kind of formal collection of “notes from the field” and solutions to common real world problems of building and productionizing ML models.
  
• Following on a bit from my point above, another reason we wanted to write this book (in addition to having a collection of common problems/solutions) was to create a shared language for these ML patterns. Like any pattern, you find yourself using the same tricks over and over. And in talking with other colleagues we wanted to create a single term or phrase that describes that problem/solution. So, for example, the “Bridged Schema” pattern gives a name to the common problem that arises when dealing with changing data fields. This way, we can try to have a shared language of common patterns for everyone to use.

Thank you so much for the answers, now I’m more excited to read the book 🤩🔝

Thanks Aleix glad to have the opportunity to talk to you as well. 🙂
That’s a good question. It’s hard to say what the most common problem would be. I work a lot with Google Cloud customers. We typically work on engagements that span the entire ML work lifecycle. The project specific problems or patterns that arise are so use case dependent. So I guess the most recurring problems then would be the problems that relate to initial or final stages of the lifecycle (since they are broad and similar across multiple use cases). Those would be the patterns addressing data issues or dealing with Responsible AI and working with business stakeholders. Typically the most common problem is in framing the problem and understanding issues with the data. That seems to be consistent across all projects I’ve worked on.

hi Matthew Emerick! Happy to be able to stop by 🙂 I know too well about large book to-read lists lol.
The patterns aren’t arranged by difficulty. Instead we arranged them loosely with the different stages of a typical ML workflow or ML lifecycle. So we start with patterns related to Data Representation, then move to Problem Representation, then talk about patterns related to Resiliency and then Reproducibility and end with Responsible AI and stakeholder management.

I like that. Is there anything about the complexity (ie bubble sort is easy to do, but slow) for each pattern?

kind of! Each pattern is broken down into four main sections: Problem, Solution, Why It Works and Tradeoffs & Alternatives. The Tradeoffs & Alternatives section discusses exactly that. We touch on any ‘gotchas’ or complexity issues, or common pitfalls or just other related approaches you might want to consider.

Okay, so the expected style for a pattern book. Makes it easier anyone who has used a pattern book in the past. Good plan. Organization is something I touch on in my book reviews, both macro and micro. A well written book with great and important content can hit a wall if it’s not organized well. Great job!

hi George Melvin thanks for having me
Good question. The patterns are more focused on implementation. So I can see how they would apply to both smaller/larger orgs. That being said, for some smaller orgs or smaller teams there is less focus on some MLOps aspects. For example, we have a section on Feature Stores. This is a super useful pattern and important to know about; however, I can image for a smaller org the need for a feature store may be less imperative than for a large organization where you have dedicated teams to engineer features for the models that another dedicated team is building.
But, I’d say the patterns in general are relevant for small or large teams and orgs. Just some may resonate more than others.

Great, thanks for your answer! I ’m looking forward to learning more 🤓

Hi Neal Fultz and thanks!
Ohhh that’s a good one. A lot of things come to mind. I think the most dangerous (and I may be biased in the type of work I usually do and some bad past experiences) would be anti-patterns related to reproducibility and resiliency. So much of ML development is experimentation (with data, with pipelines with models, etc) and reproducibility is crucial to running thorough experiments. Also, as I work with more companies interested in productionizing their models, anti-patterns related to resiliency really end up being more and more detrimental

hi Dr Abdulrahman very happy to join the channel this week.

1. I’d imagine that for most senior ML practitioners would be aware of the majority (if not all) of these patterns. Though they might not know it by the name we used. We have code for each pattern and use a variety of tools, some of them more recently developed than others. So even senior practitioners might be able to learn something new from the book.
   
2. That’s a good question. I’m not sure. I imagine it would likely come from constraints either in their infrastructure or project timing. For example, someone may understand the importance and need for Continued Model Evaluation but it can be difficult to implement carefully and require too much overhead for the business objective.
   
3. I wouldn’t say a complement book is needed, but it could certainly help. This book touches on patterns across the entire ML lifecycle so a book more focused on one aspect (say data engineering or ML pipelines) could be helpful for more implementation details.

Hi Simon Steinkamp thanks for joining with questions 🙂 I’m not familiar with the term “copy-paste” data scientist, but I think I can guess what it’s referring to. In some sense, the ideas are similar. However, these design patterns are more like ways of thinking when designing solutions or building ML systems. This book would follow a course on how to effectively build ML model, so we don’t get into the weeds about those details. There are already really great books out there that address that. The idea of patterns is more focused on ways of thinking and recognizing common solutions to common problems. The solutions can’t typically be copy-pasted directly (since each case is a bit different) but the idea is certainly repeated.

Great, thank you for your answer, I will have a closer look 🙂

hi Poornima Thank you! I hope you enjoy it.

1. The motivation came from lots of customer engagements at Google Cloud in our roles. When designing ML systems or thinking about common ML approaches, we saw that there were some common patterns in the field. The book attempts to collect those pattens into a single easy to reference source.
   
2. There wasn’t any monolithic use case or project but we did try to use similar datasets throughout the book when illustrating a concept. This way, the code solutions would be less about the data or model and more about the pattern itself.

Thank you much Mike Munn for answering. Looking forward to gain great knowledge from the book. Hoping to see more such excellent works from you in future. All the very best 🙂

Alexey Grigorev yeah for sure! It’s like the typical hammer/nail scenario. For beginners, I think it would be helpful to first just read through the patterns to be aware of what’s out there. Then revisit certain patterns based on the individual use case and really question of it will help solve the problem you have. The final chapter of the book gives an overview of all the patterns, breaking them down in a simple chart of problem/solution/description. If you think a pattern might be a good solution, take a critical look at the section that discusses that one in detail. And see if it fits or not.

Makes sense, thanks!

Thanks Justin Neumann great question. The Reframing pattern is an interesting one. It’s focus is less on the model itself and more on how the labels are represented. So, for example, the regression -> classification reframing discusses how you can modify a typical regression problem and frame the problem as learning a probability distribution via classification and binning. One scenario where it is useful is when there may be training examples which have the same features but different labels (I think in the book we used the dataset of birthweights). This way you are learning a probability distribution for a feature input and it can be less confusing for the model. So you’re really just changing your labels, and you would modify your gradient boosted tree to be classification instead of regression. The pattern isn’t model dependent. Does that make sense?
One last point to mention is that this approach of reframing may just not be beneficial to your data or use case. As with most solutions, it’s very use case dependent.
Lastly, I’m glad to hear you have the book. I really hope you enjoy it!

Thanks for the reply. Your answer makes all sense to me and you’re right, this pattern may just be not applicable to my problem. I’ll keep reading! 😉 Best, Justin

Hi Rosona Thanks for joining in. Our target audience was someone that knows the basics of ML and building models. We kind of pitched it as a follow up to a beginning ML book, like Geron’s book “Hands On ML with SciKit and TF” or Muller/Guido’s book “Intro to ML with python” or after a semester course in ML. With more experience, I image the reader would be able to appreciate (or recognize) more of the patterns.

Alexey Grigorev yeah good question. When I think about applied machine learning or ML projects we’ve done with our cloud customers, I think the most useful and common patterns are those that show up in our chapters on Repeatability and Resiliency, patterns like ML Pipelines, Continued Model Evaluation, Batch Serving, Keyed Predictions, Batch Predictions, Feature Store, Transform, and Model Versioning. Also, the Responsible AI patterns are also really important for applied machine learning projects.

Thank you! And what is the least used one? You mentioned feature stores as something that smaller teams/companies usually don’t need - maybe this one? or some other one?

yeah some of the patterns are more use case driven, so I’ve personally see those less. And yes, patterns like Feature Store seem to be much more important for industrial teams. In my experience, smaller, more agile teams might not benefit or need to have a feature store set up. They can be incredibly useful and necessary for some situations, but can also require a lot of technical overhead.

Got it, thank you!

Hi Rishabh Bhargava thanks for joining with questions. It’s funny, that question has come up before, and honestly I think we were able to catch all the patterns we had in mind. Our initial list didn’t come out to such a round number of 30, but as we were writing a couple topics we thought would be covered as one pattern ended up getting their own section, or the opposite (some individual patterns ended up getting grouped together naturally). So in the end, we covered the ones we wanted….though, now that it’s been put in my mind I’m already thinking of new patterns we could have added 🙂

I have a 2 parter question as a follow up.

1. Do you think these design patterns will need significant updates in the near future, or these patterns represent some general truths about how ML systems should be built?
   
2. If you think updates are likely, do you think the second edition will be needed in 6 months, 1 year, 5 years or 10 years?

yeah good questions,

1. i don’t think these patterns themselves will need much updates..though, for some, the code or implementations could change over time and be updated or improved as tools continue to develop as solutions to address these common problems
   
2. I don’t foresee the need for an second edition soon, but if code updates do become necessary, they’ll make it into the book’s github with the code examples much sooner.