Machine Learning Zoomcamp: Free ML Engineering course. Register here!

DataTalks.Club

Articles Slack Events Podcast Books Courses

Product Management for Machine Learning

Season 6, episode 7 of the DataTalks.Club podcast with Geo Jolly

Links:

Did you like this episode? Check other episodes of the podcast, and register for new events.

Transcript

The transcripts are edited for clarity, sometimes with AI. If you notice any incorrect information, let us know.

Alexey: This week, we'll talk about the AI Product Manager role. We have a special guest today, Geo. Both Geo and I studied on the same program, which was information technologies for business intelligence. I did this a few years earlier. After graduating, Geo worked as a data scientist, like me. This is also what I did after graduating. But after working for a couple of years as a data scientist, Geo switched his focus to product management and this is what we will talk about today. Welcome Geo. (1:13)

Geo: Thanks a lot, Alexey. Appreciate it. (1:42)

Geo’s background

Alexey: Yeah, thanks for coming. Before we go into our main topic of the AI/ML product manager role, let's start with your background. Can you tell us about your career journey so far? (1:44)

Geo: Yeah, definitely. I think that will help the audience as well. My background is mostly computer science engineering, which is what I studied for my Bachelor's. I think most of my initial interactions were more Lean, like pure software development and web. Even during my studies, I was doing a lot of websites, blogging – I was into these kinds of things. At that time, content writing wasn’t like blogs, since blogs didn't exist. It was mostly typing content. I was into that, so I did WordPress, PHP, baseline optimization, and social media marketing. I did that quite a lot, I would say. But it was good. When I think about it now, it has actually helped me to understand a lot of key concepts and basics. It was also time for me to practice what I was learning as well. (1:56)

Geo: After my graduation, I worked as a developer as well – Java development backend. I did that and then while I was working, this opportunity for BI came along. Then I moved to Europe from India and I did a Master's. After finalizing my Master's, I worked as a data science engineer. During my Master's, I worked with ING Bank for a while, but it was more like a small internship. But I think that would be my first formal experience as a product manager. I was working very closely with the product management team. After my studies, I went back to data science. So I was like, “I think this is what I want to do.” That was my impression. But then I slowly realized, “Maybe product management is something that I would be more interested in doing.” (1:56)

Geo: I always like to see the bigger picture of things and adding more value to the business. I know that data science also does it, but sometimes it's just not visible. I liked all this stuff that product managers were doing, so I slowly switched into that direction. Then I moved to the UK and was working as a product manager for Revolute for two years. Now I’m at Glovo as a technical product manager. That's my career – nothing very surprising. Normal track of software engineer to data science and some of it was continuous data science, but I decided to go into product management. (1:56)

Alexey: Yeah. That's interesting. You said that you had a bit of product management experience as an intern. But then you thought “Okay, since I'm doing this Master's, I want to do data science.” So you tried a bit of that for a couple of years and then you realized that product management was for you, right? (4:44)

Geo: Yeah, but I wasn't sure. I knew that I liked product management, but it's not an easy job to get into. Product management is not something that is taught in any university or anything like that. It's something that you need to learn by doing. So it wasn't easy to get into a product management job. I was just waiting for the right time and right moment to come along. It was there in my mind, “Okay, I want to do this.” But I was just not 100% sure. (5:05)

Alexey: Okay. So the main driver for you was, as you said, that you wanted to see the bigger picture, and not just contribute to a specific piece of the system. You wanted to have an impact on the overall product. Is that right? (5:39)

Geo: Yeah. I also enjoyed going in and seeing the bigger architectures and how these multiple systems interact with each other. As a PM, you can also set the direction of where the project is going, which I found quite interesting. I know some people like that, while others don't – they want to focus on a smaller scope and keep improving it. That's fair. We need people like that as well. I just didn't see myself doing that. (5:54)

Technical Product Manager

Alexey: Can you tell us more about what Glovo does and what you do there as a technical PM? (6:28)

Geo: Sure. Glovo is originally from Barcelona and was founded in 2015. Right now I'm based in Barcelona as well. It's a super app, but its main focus is on food delivery. The mission is very simple – to give access to anything in the city. So it really doesn't matter if it's food, groceries, pharmaceuticals, supermarkets – Glovo is set to deliver everything. The model is the traditional flywheel marketplace model. You have different sellers trying to sell their stuff, like restaurants, pharmacies – that's the model. I think almost 100 million orders are done every year, on average. The app is currently operating in almost 20 plus countries globally. (6:36)

Geo: My job in Glovo is to lead the team that’s managing the machine learning platform. Essentially, I'm a technical product manager there and I define the vision and direction that this platform should be going. That involves helping data scientists deploy different machine learning models. In Glovo, we have 50+ data scientists and a lot of data analysts. All of them are using machine learning to add value to the business. What my platform is doing is essentially helping them by providing the infrastructure to productionize their models faster and providing different tools for managing the quality of models, etc. Basically the job involves identifying what to do with the platform. “How can we improve the platform?” And “What's next? “How can we improve it and align it to make it better?” (6:36)

Building ML platform

Alexey: You said, “Lead a team that manages the ML platform.” Does this mean that you're building your own platform, or you have some other platform that you manage? (8:30)

Geo: We are building our own. It's an in-house solution. Essentially, we have a legacy platform right now. Maybe legacy is maybe a bit of a harsh term to use since it’s just three years old. But the thing is – industry is going so fast and the requirements are super high now. For example, the platform we have right now has a training orchestration that was done in Jenkins. But we are now slowly moving into Argo and Airflow, so the orchestration is more scalable. So these are the kind of things that we do. Slowly, we are improving. In the end, it's an in-house platform, though we do integrate third-party tools based on the requirements. It's not like we build everything. It depends on VSS. If you feel like, “Hey, this capability may not be feasible for us to build in-house.” What we do is we assess the vendors and say, “Okay, maybe let's use that solution and integrate with the platform.” (8:41)

Alexey: Is this something that you do as a technical product manager? (9:44)

Geo: Yeah. Essentially, my role would be to take ongoing feedback from data scientists. I discuss it with them – I may not speak to every data scientist, but I usually speak with data science leads or managers of data scientists – I take their feedback and see what the problems and challenges are in using the platform. That's one way I approach it. The second one is reviewing the existing product and seeing what's out there on the market and where the product stands. Here, I try to understand, “Hey, my platform has these gaps. How can we fill them?” This means the product needs more capabilities in the platform. (9:50)

Geo: So you come in and write the specifications and roadmap and ensure that the deliveries come from this. That's another direction. Finally, as always, you have the technical depth which may slow things down, but it can also be an accelerator. So how does one manage this efficiently and prioritize it? You need to find the right balance so that the business is not slowing down. You give the data scientists what they want. Some models may need custom tools, so you give them that and prioritize what they want. Meanwhile, you also continue improving the platform. (9:50)

Working on internal projects

Alexey: Interesting. So this is an internal product, meaning that you are not a customer-facing team, right? So what is the main idea behind your role? Why does the company have a product manager for an internal team? (11:09)

Geo: Yeah, that's a very good point. Even though it's internal, in the end, there are 100+ people using this platform. They are the customers for this platform. We need to accelerate the quality of the models – we need faster time to deployment. Therefore, we are actually enabling users of the platform to add more value to customers. It's not a direct correlation, but rather an indirect one. You are actually enabling everyone in the company to deliver more and more value to the customer. That's one way of looking at it. (11:24)

Geo: Secondly, even though it's an internal platform, you still need to understand what your customers need as the platform grows. You also need to look into the industry and say, “Hey, this is what we need to do right now. We need someone who can actually prioritize what needs to be done and what capabilities need to be in the platform.” So that's how a product manager adds more value there. Then, since it's a big platform, you have different requirements coming from different stakeholders – data scientists have different needs, then business data engineers may have different needs, apart from that, you also have internal requests, “I need these bugs to be fixed.” This requires prioritization and understanding. (11:24)

Geo: Based on all the things that need to be done, a product manager assesses what can be done and what the best way to do it is. I definitely believe that even though it's an internal platform, at some point you need product managers to make it efficient and operate it in the right direction. Otherwise, I have seen platforms that reach very, very messy states – they have a lot of tools, a lot of capabilities, and no one knows how to use them and there is no direction to it all. (11:24)

Alexey: So that's what happens when an internal team – a tooling team or a platform team – doesn't have a product manager, right? (13:37)

Geo: Yeah. Usually what happens is that developers may say, “Hey, this tool is cool. Let's integrate it.” They integrate it. In the end, you will have a platform that has a bunch of tools or libraries, or even multiple UIs. A person comes in to use it and they’re like, “Hey… Where should I even start?” Even though there is documentation, they will say “Hey, there is this tool for tracking. What about this tool?” There is a lack of consistency and a lack of UI. So in the end, what happens is – if you think about it, a data scientist’s salary is maybe 4000 euros on average – if the person is struggling to use this platform for two weeks, you're actually costing the business money. Let's take an example. Say two weeks is around 2,000 euros and there are 50 users there. You're actually losing around 100,000 euros. If you think about it like that, then you will see why you need some product managers to fix this. (13:44)

Prioritizing the backlog

Alexey: How do you help to actually do this? You said you prioritize things, collect requirements, and the requirements come from different sources. So this can be something like – you saw a demo from some external tool or somebody complained to you about something. You take all this down and put it in a backlog, and then you prioritize the backlog. How do you do that? (14:55)

Geo: The prioritization process is done with the development team. We have engineering managers and usually, the product managers work very closely with engineering managers and the development team itself. The product managers try to define what the problem is, while the definition of the solution happens with the engineering team. All these discussions are part of our weekly grooming of the backlog. The product manager runs the sessions. For example, every week, we have an hour where we discuss the top priorities. (15:19)

Geo: Then we decide together with the team by following some Agile methodologies or something like that to make sure we have deliveries. Once we agree on the backlog and time of delivery, we communicate back to the stakeholders, “Hey, data scientists requested this feature.” I will go back to the data scientists saying “We are actually working on this and you can expect this to be delivered in two weeks or four weeks or a month.” (15:19)

Defining the problems

Alexey: Interesting. You mentioned one important thing – that your job as a product manager is to think about problems. So you define a problem? But then, you're not thinking about the solution, the engineering team thinks about the solution. Is that right? (16:27)

Geo: Well, it's actually quite a complex topic to discuss. The danger is that often, people don't really understand the problem first, and they go directly into solutions instead. That's very dangerous, in my opinion. I have seen examples where people said to me, “Hey, I want to solve recommendations and multi-category recommendations will work. I read an article in Medium about it.” You can draw conclusions right away and say “This solution may work.” But it is also important that you validate your hypothesis, etc. As a product manager, I say what the desired outcome would be, but not the solution. (16:44)

Geo: Let's say I want the model training time to be reduced by X percent. Now the outcome is clear and the metric is clear. While the engineering team I work with will have to go back and forth. Now that the problem is clear and the outcome is clear, we just need to find how we reach that stage. It's always important that you define your problem clearly and decide what the outcome should be. The outcome is essentially how you will measure it. I think it's easy to say, but I spend a lot of time doing this. (16:44)

Observability metrics

Alexey: So, you mean KPIs and things like that, right? How can you measure that your platform is good? How do you measure it – if you reduce the time that a data scientist spent with your platform? It's very hard to track, right? (18:12)

Geo: These things are hard to track, so you have to build observability metrics. You need to push more metrics to maybe Datadog or whatever observability tools you're using. I would say this is an interesting topic that even industry is hard-pressed to answer sometimes. Because I think many of the ML Ops tools in the market don't have this answer yet. It's a problem I think everyone is trying to solve. (18:25)

Avoiding jumping into “solution mode”

Alexey: Yeah. The question I actually had when asking that is – I know that you have an engineering background. You spent some time doing Java development and you spent some time doing data science. Maybe not now, but some time ago, you were quite hands-on. You still know how to do some things – you still remember and have this experience. So how do you make sure you don't jump into ‘solution mode’ and that you don't start recommending teams what to do as a PM? (18:56)

Geo: Yeah. That's actually a very good question. It's also one of the hardest things that you struggle with on a daily basis as a PM. I think it's the mindset of a PM where you need to actually think more about the problem itself. Sure, I think I want to jump into solutions, but mistakes are usually made because you can sometimes get too excited about a problem. But it's not like, “Stop being excited.” It's just about spending more time on understanding the problem itself and breaking it down into small pieces. That's one exercise that helps prevent me from going directly into solutions. (19:29)

Geo: When you start breaking down your problem into smaller pieces and figure out how to solve it, you essentially understand that, “I cannot think about one solution directly. It's not the right way and right time for me to do it. I should try to spend some time clarifying this problem and breaking it down into small pieces.” That's one way of doing it. The other one is that it just takes practice and consistent effort. Sometimes you do that as well. Then you will have to be realistic about it and say, “I did this and I need to make sure that I'm not doing it anymore.” (19:29)

Breaking down the problem

Alexey: Yeah. Interesting. You said you spend a lot of time trying to understand the problem and break it down. Do you do this alone or the whole team is involved? What does the process usually look like? (20:55)

Geo: Yes, sometimes. The whole team is involved In some of the cases. It’s mostly with the actual users, which can depend on the job I do. In the previous job, it was sometimes with regulators or compliance managers and some customers. Right now, I mostly spend time with data scientists and take their feedback, do some brainstorming sessions. It may be just a small session where we do an exercise together and say, “Let's go deeper.” We list out all the problems and group the problems together. Usually, this involves workshops or customer interviews, or user interviews and things like that. This helps us to better clarify the problem. For the most part, I would say that we talk – communicate and understand each other. That's the way it usually works. (21:06)

Important skills for product managers

Alexey: Would you say speaking, communicating, and prioritization are the main skills you need to have as a product manager? (22:05)

Geo: Ideally, yes. I think those are the key drivers. As a product manager, most of your schedule is built around communication, so you spend a lot of time in discussions with people. Good communication is a very important skill to have. I also don't believe that everyone has this skill from day one. It takes time and practice. Prioritization is a key skill as well, which I would say is also not an easy thing to master. I know people with 10 or 20 years of product management experience and they still spend a lot of time developing this skill because it's not easy at all. You have many people with many requirements and all of them are important. So your job is to go back and be realistic in prioritizing things – What needs to be done? What brings value to customers? (22:15)

Alexey: What other kinds of skills do you think you need at work, in addition to communication and prioritization? (23:19)

Geo: In my specific case, since I’m closer to the ML world, I would say it's also important to understand the market or where the industry is going with machine learning. This includes machine learning ops and engineering practices, cloud technologies, etc. You need to be constantly updated, because you may be solving the problem in a really wrong way or maybe you're not using the right tools. To avoid this, you need to be constantly learning about what's happening in the industry. Maybe this could be a review of some materials or participating in some events. Learning is very important, I would say. (23:28)

Geo: Secondly, I think you need to have an understanding of model architectures and other technical things, like building event streaming systems or Bats or Big Data-related knowledge and database-related knowledge. All this can help because, in the end, your communication is based on these topics. If you cannot understand or you cannot stay updated with it, you are going to struggle a lot as a product manager. You need to be really updated with what the technologies are in the company and how people are using them. (23:28)

Geo: There could be security-related issues from not knowing these things or it can be a wide range of other issues, so you need to have a good understanding of how it works. Otherwise it can turn into a bit of a disaster – being a product manager who communicates a lot but doesn't understand what's happening under the hood. For that, learning is quite important, I would say. (23:28)

The importance of a technical background

Alexey: How important do you think it is to have a technical background for a technical product manager? I’m guessing you studied all the things you mentioned and you probably also know them from your work. (25:16)

Geo: We have different kinds of product managers and we can go into that later. But in my kind of role, I would say it's very important. If you're communicating with data scientists often, you need to understand things like the lifecycle of the model and how the model works. You need to have an understanding of algorithms, etc. So that's very important. On the other hand, you're also going back and talking to your team – which is essentially engineers who are building a platform – so you need to understand different cloud technologies, tooling, and Kubernetes (or whatever infrastructure solutions that you're using). You need to have an understanding of this as well. Some of the technologies I had to learn on the fly, because in Amadeus, the cloud platform we used was Redshift, which was a private cloud that is managed by Amadeus itself. It was like OpenShift and etc. That’s something you learn there. (25:31)

Geo: Then when I went to Revolute, it was fully on GCP. I was new to GCP at that time, and you just learn the concepts and understand what GCP offers, which could be dataflow, Pub/Sub, etc. Then you come to Glovo, where we are primarily using AWS. Now I'm in a new world of AWS. So it's just a matter of what you’re doing. I think the underlying concept is the same, but you just need to be really aware of what's happening in all of this as a technical product manager. Otherwise, I think you will struggle with prioritization, because you won't understand what to solve or you will struggle with how to solve it. When solutions are being discussed, you will feel like, “Hey, what's actually happening? I don't understand anything.” In very platform-specific technical roles, I would say, yes – it's very important to have technical knowledge. (25:31)

Data Lead vs Staff Data Scientist vs Data PM

Alexey: Thanks. We have quite an interesting question. What you described is, to some extent, quite similar to what I do. Although I'm not a product manager, there is some overlap, like discussing architecture, talking to data scientists and seeing what their problems are. I'm on the individual contributor track of data scientists, so the title is Principal Data Scientist. And the question that we have is, “What is the difference between the Data Team Lead and a Data PM?” In this case, I think the question is very similar to “What's the difference between, let's say, a Lead Data Scientist and Technical AI Product Manager? (27:29)

Geo: Yeah. I think lead data scientists are also managers. If I understand the question correctly… (28:15)

Alexey: Not always. This is more like – you can think of it on an individual contributor level – after Senior. In some companies it’s called a Staff Data Scientist. (28:28)

Geo: Okay. Now I get it. I also work with Staff Data Scientists and Staff Engineers as well. The main difference is that they don't drive a roadmap. Or rather they may drive one of the items in the roadmap. For example, they may own a work stream or multiple work streams. I think they come in a lot when we are defining the solution, like I mentioned earlier. Product managers are usually the ones helping define the problem and the outcome, while lead or staff engineers are the first people who usually come to help me define the desired outcome, or the solution itself. So that's the first main thing. (28:37)

Geo: The second one is – I don't really go into the details of the quality of code. I don't go into the details of going into reviewing pull requests on a daily basis. I don't go into any technical architectural decision-making. I do say my input – that's always welcomed – but it's not my expected responsibility to go and make this decision. I think this is what’s usually done by the data science leads or staff data scientists. (28:37)

Geo: Then there’s also the model architecture. What I will be interested in is mostly the results. “What was your hypothesis? How did you test it? What's the outcome of your A/B tests? How did we add value?” That would be what I would be interested in rather than saying “I chose a deep learning model” or something like that. In terms of this decision making, I am only interested in the outcome, while the lead or the staff data scientist is going to actually work on the architecture and understand how I translate these problems into actual requirements and work on it. (28:37)

Alexey: Actually the question was the difference between the Data Team Lead and the Data PM – I think I slightly rephrased it because I was more interested in hearing the difference between individual contributors and product managers. But let's say that this person is a data science manager – they are the Data Team Lead or whatever you want to call them. So what's the difference here apart from having to manage people? (30:50)

Geo: You said if it's a Data PM, or just a Data Science Lead? (31:16)

Alexey: Let's say the difference between a Data Science Lead and a Technical Machine Learning Product Manager. (31:21)

Geo: Sure, definitely. Usually data science leads will have ownership of multiple scopes within the same product. Let's say a product is for recommendations and these recommendations may have multiple types. This is a bigger scope. What I've seen is that the data science lead would own the full direction of this project, while the product manager is very much focused on the customer side, like “Where will this recommendation be placed in the UI?” The product manager also works with the UX or designers to design what the display will look like. There is also backend that’s involved in this model process – you need to integrate your models into the backend. The product manager acts as the center point of contact for that and ensures that the collaboration is happening effectively with backend and data science lead and their data scientists as well. (31:28)

Geo: Importantly, the product manager also defines the rollout strategy. Let's say I have a model that is performing really well. When should I go to the market with it? Do we need to do any marketing campaigns for it? This is defined by the product manager. While the data science lead’s input is always welcomed, it's not something that they actually do on a daily basis. Sometimes governance is a big factor – data science models that need to be regulated and need to undergo an approval from a governance board. The product manager is actually the one who collects the results with data science leads and data scientists, and can go to the governance board and say, “Hey, these are the results and this is the outcome. Can you please approve it?” This is the business use case. These kinds of external communications, approvals, governance – that’s all monitored by the product manager and not data science leads. (31:28)

Approvals and rollout

Alexey: It seems like product managers have more responsibilities here because this is them saying, “This should be approved.” Especially when it comes to regulations, the liability or responsibility is much greater. It’s ultimately the PM’s decision to go with a particular model. (33:47)

Geo: Exactly. Essentially, it can really affect your regulatory licensing, or it can be money laundering regulation, or other things like that. This can potentially lead to huge fines, and other negative circumstances, like what happened with Zillow, that everyone is talking about now. But it’s similar for banking, it can happen as well. But I think in financial tech, I have seen these deeper regulations and governance being put in place because of the nature of the business. For Glovo, this hasn’t happened yet. I don't think it's that regulated. There is regulation for a lot of things, but it's not a very strict governance process for model release as of yet. I think people who are working in FinTech can relate to this more. (34:10)

Alexey: Also as an internal PM, maybe you need to deal less with these kinds of things. You mentioned thinking about time to go to market and about marketing. I’m guessing this is out of scope for you right now. (35:01)

Geo: Right now, yes. But I would still say that some of the features and capabilities that we release, it is also important for me to understand who will adopt it and how many people will use it. It's not feasible for everyone in the company to adopt the capability of the platform, so we need to understand who is going to adopt it and when. So this is a sort of “time to market”, or rather “time to stakeholders” where we need to consider how we roll out and adoption. Therefore, you need to align that with them as well. (35:18)

Alexey: So it's similar. Maybe the market you have is slightly smaller, but you still have a market, right? (35:56)

Geo: I would say it’s negligible compared to millions of users. But I think these people are the hardest because everyone has a say, since they speak to you on a daily basis. But with a customer group, you actually may just release it to beta and go into millions of users right away. There are pros and cons in both ways of thinking about it. (36:06)

Engineering/platform teams

Alexey: When you say “engineering team,” do you include data scientists? Or are you talking mostly about backend developers? (36:36)

Geo: When I say “engineering team,” it's only the backend engineers right now, even for the ML ops platform that I'm doing. We do have two data scientists as well. Currently, they are part of the engineering team, because I'm still structuring the teams there. At some point, we want the data scientists to be in an independent team. It's an exceptional scenario right now – that the team that I'm working with has two data scientists embedded into the engineering team. Ideally, in other teams in Glovo, the engineering team is usually just the backend engineers or system engineers, development engineers, and data engineers. Often they don't really have any data scientists as part of the engineering team. (36:47)

Alexey: The team you work with directly – the team that builds the platform – who are they? What kind of responsibilities do they have? What are their roles and titles? (37:40)

Geo: We define the user journey of the data scientists starting with offline experimentation, such as what notebooks are offering infrastructure for this. Then you have data management, which is a huge area. You need to integrate with the data lake, data warehouse, feature store, etc. For this data management-related tooling, we have the backend engineers building tooling around it. Then, some of the data quality-related tooling, like Great Expectations, integrating these libraries with our platform – this is done by backend engineers. (37:48)

Geo: Then we have model training infrastructure. We allow data scientists to schedule workflows for model training, which are essentially model training jobs that can be scheduled and run on a daily basis or weekly, based on the business requirement. This has underlying infrastructure, which is Kubernetes, because we are using Argo right now. This is managed by system engineers, or people with Kubernetes knowledge. Then once the model is trained, you need to go into the deployment phase. (37:48)

Geo: The model needs to be productionized, so it could be served via an API, or it could be a basic consumption of key value store. It's a batch consumption of reinforcement learning, based on the approach. All this requires CI/CD pipelines, which is probably Jenkins, plus we also have Spinnaker for it. All this is managed within the platform and any bugs in this are fixed by these engineers. If something breaks when a data scientist is trying to deploy something, we have an internal Help Center, in which they raise a ticket to us saying that they have an issue. If it's a high priority, we prioritize it and try to fix it. So it's a mixture of system engineers and backend engineers. We don't have a service, but at some point, we would like to have one to maintain the infrastructure part. (37:48)

Data scientists’ role in the engineering team

Alexey: You mentioned that there also are a couple of data scientists who would take care of offline experiments. Did I understand it correctly, or are they more like users and not necessarily in your team? (40:03)

Geo: We have a couple of data scientists, but they don't actually own any particular parts of the user journey right now. They are very focused on tooling, such as data science tooling, A/B testing tools, data quality tools. We also have a core library of the platform, which they manage as well. The core library of the platform also has integrations with Standard BS libraries, like NumPy and Pandas. All of this is managed by data scientists. (40:14)

Geo: At some point, they also act as accelerators, like power users of the platform. Let's say another data scientist is joining the company and he has a new model. When he tries to use our platform, these data scientists are more like consultants to him saying, “Hey, this is how you should do it.” They will help him with the best practices in the platform, and sort of guide him through using the platform in an effective manner. (40:14)

Alexey: Like DevRel advocates, right? (41:25)

Geo: Yeah, exactly. They’re more like developer advocates, but they also do some modeling for internal purposes. We have some dummy models that we use. For instance, recently, one of them did a model for predicting how many Jira tickets we would get on a weekly basis. It’s a very simple model, but it's just used for us to manage the lifecycle of the platform. We can use this for tutorials and writing demos, etc. (41:28)

Scrum and Agile in data science

Alexey: Interesting. You mentioned Jira tickets and then there is a comment that says Scrum doesn't work for data science projects. So what do you think is the best way to manage data science projects? (42:01)

Geo: There is no best way, in my opinion. I don't think that there is a singular answer to this. I think it's very subjective to the team, the company, and the culture. You need to try different methodologies accordingly. When I started in the role, there was nothing like that. My job was to build a team, so we weren't doing any Scrum or anything like that. We just had a to-do list. Then we reached the stage where we had a few people, so we decided to maybe do Kanban. We just had a high priority list and we just did the tasks on it. (42:14)

Geo: Then we reached the stage, “Okay, now we are in governance. We need to make releases. Okay, let's do Scrum.” So I think it's an iterative and adaptive process. I personally don't think it's good to say, “Hey, this is not going to work” without trying it. Scum has been around for maybe 20 years or so. It's used often and it can work. I know that in some places it doesn't work. So you need to understand what the needs of the team actually are. Maybe some teams are research-oriented, like R&D teams, they may not need Scrum. It depends on the scope of the business, where you are, and how you want to deliver. That's my personal take on this one. (42:14)

Alexey: I haven't worked in Scrum in like five or six years. But what I remember as the main problem with this is that data science projects are usually not very deterministic. Sometimes it's hard to say, “Okay, this will take 5 story points or 20 story points.” So I guess that's the main problem. Then at the end of the sprint, you say, “Yeah, that didn't work.” (43:46)

Geo: Yeah. I understand what you mean. It depends. I'm just saying that in terms of Agile practices, you can try Kanban – then it may work. You don't need to size all the points. That's why we are retrospectives. We speak with the team to understand what's working and what's not. Product managers should also do this because in the end, it's affecting the delivery. You need high quality delivery and everyone in the team should be satisfied with it. Product managers have a very important responsibility to do that as well. (44:13)

Transitioning from Data Scientist to Technical PM

Alexey: Thanks. I'm quite interested in your career journey, or rather in your transition. When you decided, “Now I want to have a larger impact. I want to do less hands-on work and focus more on defining problems rather than solutions.” So from the moment when you realized this to the moment when you got your first job offer as a product manager, what did you do during this period? What did this transition period look like for you? (44:56)

Geo: I actually started being into product management maybe three or four years before I started my first job. I was in different communities in product management. I read a few books about product management. I was constantly reviewing talks, which were something like “product school.” I think it's very popular now. They were doing something like what you are doing for data science, but they were doing it for product management. There were a lot of product managers coming from Google, Facebook, and speaking about product management. (45:26)

Geo: I also closely worked with a few product managers in order to understand how they are doing the solutions and everything. Slowly, I realized that I already did a lot of product management when I was doing my blogging – when I was maintaining my own website. I realized slowly, “Hey, I actually did some product management before because I was building a UI which is used by quite a few people. I was trying to make revenue out of it by placing ads in the right places.” Even in the data science lifecycle, you spend a lot of time understanding the business, and therefore you are actually doing some amount of product management already. It’s a matter of how you take it to the next level. (45:26)

Geo: I just translated everything I did into a product mindset to say that, “Hey, I built websites that were having X number of visitors and that generated X amount of revenue.” I just updated my CV with that. Then I thought about the data science projects I did. For instance, I was working for a chatbot project. I was collecting the requirements, speaking to the client all the time because there was no product manager. Then I was leading the product for six months in India, and I delivered the output as a chatbot. Then I took feedback from him. I built metrics to monitor it, like, “What was the satisfaction score of the chatbot? What was the sentiment score of the chatbot?” So when you think about it, I actually did some product management there. (45:26)

Geo: All these projects, I correlated into something that was more like product management. I knew that I applied for a job that I was not really qualified for, because I was certain that I was under-qualified. But I still gave it a try. At the time, Revolute was quite open about these things and they wanted someone with a data science background. This was actually quite a hard combination to find, so I had plus points there going in. And it helped. (45:26)

Geo: It took me some time to prepare for it and to translate everything into a product mindset – a lot of reading, speaking with a lot of people, and understanding it all. As I said, you cannot learn product management just by reading, you need to do it. So I did some projects as well, which helped me to communicate that I was doing X and delivering value. I think that was a simple process. I mean, I can't really say simple, but that was the process. (45:26)

Alexey: Okay. So basically, as a data scientist, you were already in quite a good position to move more into the business side of things – into the product side of things. As a data scientist, you have to do these things anyway. You have to understand your domain and work closely with product managers. Then you can just do more of that, right? Eventually, you partner with your product manager and maybe take part of the responsibility. That’s the way to do it? (48:58)

Geo: Yeah, exactly. That's how I transition. I was doing exactly the thing you described. (49:29)

Books to read for the transition

Alexey: You also said that you took some time to prepare and that you did some reading. Do you remember what kind of reading you did? What did you read? (49:36)

Geo: Yeah. There are books called “How to Crack a PM Interview” and “How to Crack a Coding Interview.” I read the coding interview one when I was applying for engineering jobs. After that, I read Cracking the PM Interview. I think there are a lot of materials out there about product management, especially those that come from product managers themselves, because all these scale-up startups are doing a lot of stuff. I think “How to Crack a PM Interview” was one thing that helped. (49:43)

Geo: Then there's also “The Lean Startup” book by Eric Ries. I think that mentions the importance of measuring things and going in with a lean philosophy, which is a very key value and important in product management. The other thing is, I also constantly listened to a lot of people speaking. For instance, as I mentioned, things like not jumping directly into solutions. (49:43)

Geo: Another important thing is – you also shouldn't make a lot of assumptions in product management. These are the kinds of things you learn by doing and by listening to people. We are wired to make assumptions, but I have seen cases where one wrong assumption can turn into a lot of bad business issues. These are the kinds of things you only learn by doing stuff, observing, and speaking to people. You learn from experience as well, over time. (49:43)

Alexey: Okay. As data scientists, people should get more into pairing with a product manager, then reading “Cracking a PM Interview” and “The Lean Startup” books. And also by just talking to product people. That's the summary? (51:19)

Geo: Yeah, that's the small summary, I would say. (51:36)

Alexey: And do this for two years. Right? [laughs] (51:37)

Geo: Yeah. This is like a summary of what I did for two-three years. I think it also depends on your manager as well. Once you start in the product area, if your manager is quite good, he can be your mentor as well if he's giving you good feedback. If not, you should definitely find someone to mentor you. It's important to have a good product mentor. Also, I think the Airbnb Chief Product Officer, (I forgot his name) writes a lot on product management as well. I follow a lot of product people in my LinkedIn and other social networks. I get random thoughts here and there, which are like food for thought. That’s important as well. (51:19)

Transitioning for non-technical people

Alexey: Okay, thanks. What do you think about people who do not work as data scientists right now? Or engineers? Can they become machine learning product managers? (52:30)

Geo: It's easier for them to become a traditional software engineering PMs, I would say. Because in this role, you go into writing specifications, understanding the requirements, etc.. If they have some understanding of machine learning philosophy, yes – it will work. But a 100% immediate transition may not be that easy. However, I would never say a hard ‘no’ to it. I think some people are very good at learning. So if they can learn the philosophies and understand how an AI project is structured, I think they can do it as well. In the end, most of the things I mentioned, or the things I do as an AI product manager – I think 80% is the same as normal product management. It's just that 20%, where I add more value with my background. (52:45)

Geo: If you're an engineer, you could add this 20%, but you may need to learn a bit more about AI product management. This shouldn't be stopping you from applying into ML product management jobs. That's just my personal take. It also depends on companies. Some companies are like, “I want someone like this, this, this.” And some of them are more flexible. Therefore, you need to find people who are going to trust you and give you that kind of opportunity to join. (52:45)

Alexey: You mentioned that you applied for a job that you weren't qualified for. I guess when you read the job description, you never know whether the people on the other side will give you this opportunity to learn or not. So the only way to find out is just to apply and see what happens, right? (54:15)

Geo: Yeah. Definitely. I think that's the philosophy. Also, when I was applying to Revolute, I really liked the product and was using it for two years. So when they spoke with me, I knew a lot about the product and they knew that I had a clear passion to do something to help their business. This was helpful for me to join them as a Product Manager. I think sometimes, if you can clearly showcase your passion and your interest in this domain, I think it is possible to crack into it. (54:35)

Doing user research

Alexey: Interesting. Thanks. We have quite a few questions, but I don't think we'll be able to cover all of them. But let's try to cover at least a couple. There is a question from Eva, which reads “How do you deal with user research and the result? How does it influence your work?” (55:17)

Geo: My research? (55:38)

Alexey: Yeah, so how do you deal with user research? Do you do user research at all? (55:39)

Geo: Yeah. User research, in my case, I'm assuming is more the data scientists and how they're using the platform. Right now, what I do is speak with data scientists and data science leads on a monthly basis. We have follow up sessions with them about feedback. We frequently do surveys, like once every two months, about feedback and which capabilities are being used. Then we produce something called a Happiness Report for the internal platform. Let's say, for this platform, our customers like the features and they are using them. (55:44)

Geo: However, the complaints they usually have are about documentation and its lack of quality, and perhaps some lack of stability in certain areas. Then we come back, I share this knowledge with the team and say, “This is the feedback we got. How can we work on it?” So when I prioritize my objectives or key results for the next quarter, or even for a sprint, I try to address some of these challenges in an ongoing way. It's also an objective of mine as a product manager, to increase the usability of the platform. To do this, we constantly speak to the customers to understand what the users need. Then we translate that into our objectives and use some initiatives to keep it up. (55:44)

Quality assurance in ML

Alexey: Thanks. Another question from Eva is, “What do you think about quality assurance people? What is their role in ML products?” (57:09)

Geo: We didn't have QAs in the last two companies I worked – Typical QAs, if I understand the question correctly. In Revolute, the expectation is that the engineers, data scientists, and MLEs should do the QA of their own code. As a product manager, I had a checklist for each release, especially if my product was a governance product. There were five different benchmarks to be passed for a model to go into production. Let's say there is a new version of your model – we first tested it using historical data for maybe six months to see how the model performs. Then we took some publicly available industry benchmarks for the same use case and we validated it using that. Then we also had a labeled ‘gold’ dataset as well. After that, we shadowed it in the production with actual live traffic and looked at the results from that. (57:20)

Geo: So we had this quality control process for models that the data science team and I worked with. Every model and every version of a model we created, we tested it in this manner and slowly rolled it out this way. That was the quality control that we did at my last job. Here, in the platform team, it’s usually the engineers themselves that check QA. They write their own tests and we try to make sure that we have good test coverage and everything. So, in the end, I don't think everything is perfect, since people always have bugs. If you don't have bugs, I don't think you're doing your job correctly. [laughs] (57:20)

Alexey: Yeah, thanks. Do you have a couple more minutes? (59:22)

Geo: Yeah, sure. (59:24)

Advice for supporting an ML team as a Scrum master

Alexey: There is another question also from Eva. Thanks Eva for asking questions. “Do you have any tips for somebody who is working as a scrum master right now (someone who is totally new to machine learning) and they need to support a machine learning team. Do you have any advice for this kind of person? What can they do to support the teams better?” (59:22)

Geo: I am not sure I understood the question correctly. Usually, a scrum master, if I understand the role, is mostly about running Agile practices. This is something I also do sometimes as a product manager. If the question refers to extending the scope of this role, you just need to take more time in the problem definition part and work with the product manager to define the problem properly. As a Scrum master, you are likely already quite experienced in delivering the product. You know how to prioritize your backlog, which is a great asset for any good PM. (59:52)

Geo: Even though Agile philosophies are not directly mentioned in PM interviews, most companies value having a good Agile background quite a lot. How to prioritize your backlog, how to deliver a roadmap – all of this is very important. I also think the extension involves thinking about how to create a roadmap rather than someone creating it for you. Again, you need to work with your PM to say, “Hey, maybe for this project, I can create a roadmap. And I can deliver it.” So I don't think the transition is super hard, because I have seen a lot of Scrum masters coming into product management. A lot of business analysts come into product management as well. (59:52)

Alexey: But if you want to become a machine learning product manager, you also need to learn a bit of machine learning. (1:01:31)

Geo: Exactly, you definitely need to learn machine learning as well. Yeah. (1:01:36)

Wrapping up

Alexey: Okay. Yeah, thanks. I think we should be wrapping up. Is there anything else you would like to mention before we finish? (1:01:42)

Geo: No. I just wanted to say thanks a lot for inviting me to share. Also one thing about product management – I think it's also important that people don't think this job is quite “chill.” It's a very demanding job. You should always think about how to get out of your comfort zone on a frequent basis. Sometimes I'm doing the job of product manager, I'm doing product ownership, scrum master work. Sometimes when the EM is not there, I have to cover for engineering management. So there are a lot of responsibilities. But it's super exciting because you get to see the wider picture of the business. You can see how you can contribute and add more and more value to the business. I'm happy if anyone has any feedback or any questions. Or if someone wants to join Glovo, we are hiring data product managers. So yeah, please reach out to me, I'll be happy to help. (1:01:51)

Alexey: Is it only for people who are in Barcelona? Or you're also hiring remotely? (1:02:53)

Geo: We have remote teams, actually. We started some remote teams – we call it Project Jupiter, but it’s more of an experiment. We have some engineering teams that are remote. Glovo is not traditionally a remote-first company because it's very operations heavy. In engineering we are opening a few remote positions, but we also have tech hubs in Madrid, Warsaw, and in Barcelona. So if you're based there, it's perfect. If not, there are few remote roles as well. We are hiring a lot of data managers as well. (1:02:58)

Alexey: If somebody wants to apply or somebody wants to ask you a question, what's the best way to find you? (1:03:35)

Geo: Yeah, just LinkedIn. As always. [laughs] That's the best way. Yeah. (1:03:40)

Alexey: Okay. Thanks. Thanks a lot. Thanks for joining us today. I know it's pretty early for you. So thanks for finding time to join us and share your experience with us. And thank you everyone for attending and asking questions. (1:03:45)

Geo: Thanks a lot Alexey. (1:03:59)

Subscribe to our weekly newsletter and join our Slack.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.

DataTalks.Club. Hosted on GitHub Pages. We use cookies.