MLOps Zoomcamp: Free MLOps course. Register here!

DataTalks.Club

Home Articles Slack Events Podcast Books

From Software Engineer to Data Science Manager

Season 12, episode 1 of the DataTalks.Club podcast with Sadat Anwar

Links:

Did you like this episode? Check other episodes of the podcast, and register for new events.

Transcript

Alexey: This week, we'll talk about changing careers from software engineering to data science management. We have a special guest today, Sadat. Sadat and I are colleagues – we both work at OLX Group. He joined OLX as a software engineer, and then he got promoted to an engineering manager. And finally he decided to change his career and switch to data science management. I invited Sadat to talk more about his career transitioning. Welcome. (1:05)

Sadat: Thank you very much. Super happy to be here and talk to the community. (1:32)

Sadat’s background

Alexey: Happy to have you here, too. Before we go into our main topic and discuss your career transitioning in detail, maybe we can briefly go over your career journey so far. So how did it start? (1:37)

Sadat: Sure. Here's a fun fact that I think not a lot of people know, but I actually always wanted to become a doctor – an inspiration that I had from my father. Engineering, or maybe data science management, was really something that became a reality much later on in this journey. In adolescence, teenage years, is when I realized that I'm super curious about technology. Those things fascinated me. It turns out engineering was then a better option. (1:51)

Sadat: I have a Bachelor's in electronics and I have a Master’s in informatics (information technology) from a university in Germany, in Stuttgart. In between my Bachelor's and Master’s, I worked for a brief period in India, around two and a half, three years. To be honest, not the best part of my career. I wasn't doing something crazy. But I think my Master’s changed a lot of that. (1:51)

Sadat: During my Master’s, while I was a student, I was also working for the Fraunhofer Institute, which is a very famous German Research Institute, as a hivi (I'm not really sure what it's called in English). It’s a student job and you work on different projects with people writing their PhDs or full time employees of the Fraunhofer Institute. (1:51)

Alexey: Research assistant, I guess? (3:10)

Sadat: Research assistant? Yeah, probably. I think it's called “hilsveitshaft” or something in German, “hivi” short, but yeah, I think a research assistant works. This was fun. That's where I got my proper exposure to software engineering and also maybe my first introduction to computer vision. There was a brief project that I did with insurance companies, where the idea was to look at the image of a car and identify dents in it. OpenCV – good old days. (3:12)

Sadat: After I finished my Master’s, I moved to Berlin. Ever since I've been here, OLX is now my fourth company in Berlin. I started at ReBuy, Delivery Hero – there was a brief stint at Daimler, the company that owned Mercedes Benz – and now OLX. (3:12)

Alexey: For how long have you been with OLX? (4:04)

Sadat: It's almost three years now. In February, I'm going to be finishing my third year at OLX and starting my fourth. (4:08)

Alexey: What did you actually study for your Bachelor’s? You said electronics, but I have no idea what it actually means. (4:14)

Sadat: Neither do I, to be honest. [chuckles] It’s been a long time. (4:21)

Alexey: Did you learn to solder or what? [chuckles] (4:23)

Sadat: Right. It was a Bachelor's – it was an engineering degree in electronics, so I had a lot of signal processing. Hardware, really, that was the focus: semiconductor compositions, PCB design, stuff like that. Not a lot of computer science. Actually, I had two courses related to computer science in my Bachelor's. (4:25)

Alexey: But you learned to program there. (4:47)

Sadat: No. Like I said, that was during my Master’s. I guess I probably learned to program during my Master’s. I always had this curiosity. After my Bachelor's, I worked for two and a half years in India. During this period, I tried to learn programming, but I wasn't very good at it. Also, my job was called a software engineer, but I wasn't engineering any software in those days. (4:49)

Alexey: [chuckles] What did you do? (5:21)

Sadat: I was a glorified accountant, I would say. I learned a lot of Excel tricks. I know a lot of Excel shortcuts that most engineers probably don't. (5:22)

Alexey: Can you do a pivot table? (5:33)

Sadat: I can do a pivot table, I can do V lookups, I can do edge lookups. [chuckles] I haven’t used them for a while. (5:36)

Alexey: So that's how you started data science, right? (5:41)

Sadat: That's how. A lot of those things actually do… That's the thing, I think it's interesting. No matter what you do, it might not seem interesting, but if you're learning something, anything – keep at it. You don't know when it'll become handy. Some of the stuff that I learned back in those days with Excel, I do use today, as a manager. I work a lot with Excel now. (5:44)

Alexey: Do you work with something else as a manager? [chuckles] (6:06)

Sadat: Documentation. I think that would be one. So the Google G Suite – that is my new IDE that I have. I've started noticing that I’m more often on that than I am on an IDE. (6:10)

Sadat’s backend engineering experience

Alexey: Okay, let's go back three years. So what did it do at OLX as a back end engineer? (6:23)

Sadat: I started OLX as a senior engineer for the search team. When I was hired at OLX, I was already in the field of search (search and recommendation) for a while now – around three, four years, I guess. So I started at OLX as a search engineer. My first tasks, if I remember correctly – the day I joined OLX, my manager met me in the kitchen (in the lobby) and he said, “We are having an issue on Solr right now. Welcome, but I'm gonna come back for you later.” (6:31)

Sadat: This was my introduction. I landed on this ship and there was already a fire. Solr, as a lot of people who’ve used it, can maybe understand that sometimes you have a setup that isn't optimal and it's always screaming and shouting and burning. (6:31)

Alexey: What is Solr? For those who don't know? (7:19)

Sadat: Right, sorry. For those of you who don't know, Solr is basically an inverted index document store that is powered on this library called Lucene, which is the same library that powers Elasticsearch. So if you haven't heard of Solr, but you know what Elasticsearch is, it's basically Lucene wrapped under a different kind of image. Very similar stuff. (7:21)

Alexey: Inverted index document store means “search engine”. Right? (7:50)

Sadat: Right. [chuckles] Exactly. Inverted index document store. I can go into a lot of detail on what an inverted index is and how it works. But TLDR – it's what search is. It's what powers search for most companies. (7:52)

Alexey: So you knew this before joining OLX because you already worked for a couple of years as a search engineer and this is something you knew how to use? (8:09)

Sadat: I have worked with Solr quite a bit. I've also worked with Elasticsearch. I've dabbled a bit with a few other niche-er technologies in search, let's say. But, yeah, I'm very familiar with how these technologies work. (8:19)

Alexey: On your first day there is a fire – Solr is down – your manager is fixing it. So what do you do? (8:35)

Sadat: Thankfully, he didn’t ask me to fix it, because that would have been pressure, right? I enjoyed my coffee, met my new team, and life went on. A week later, we were writing the post mortem for what had happened. I don't remember the details, to be honest, but it was always something related to the autoscaler. Briefly, for those of you who worked with it, Solr is an interesting thing. Solr cloud is designed to scale with load and our cluster was configured to scale with load. (8:42)

Sadat: Now, whenever a Solr cluster scales out, which is adding nodes, there is a lot of CPU-intensive tasks that need to be done – which is to copy the data, propagate it to this new node, assemble the node, distribute traffic to this new node – a lot of stuff that is very CPU intensive. Through that, as a system that is already under CPU load to do that, and you have what is the classical “fireball” of Solr crashing. Because Solr wants to scale out and it cannot even do what it is already doing – and because it cannot do what it is doing, it wants to scale out even more. (8:42)

Sadat: The easiest fix is to stop that configuration (stop Solr from autoscaling out) which works all the time. That's what we also did eventually at OLX. Fun fact, our scaling is now scheduled, rather than automatic, so we do it at perceived intervals of time, rather than dynamically with traffic. (8:42)

Alexey: So, let's say that during the night, it's two nodes and during the day, it’s four nodes or something like that? (10:12)

Sadat: Something like that, yeah. Those numbers are probably multiplied by a huge number. (10:18)

Alexey: A hundred? [chuckles] (10:22)

Sadat: Yeah, somewhere around 50, maybe. But, yeah. (10:23)

Alexey: Okay. Well, what did you do apart from that? I don't know how much time you spent fighting fires. In general, what kind of tasks did you need to work on? (10:27)

Sadat: What had happened when I landed at the search team at OLX was that it was also the start of the pandemic – or almost the start of the pandemic. We didn't know that yet. There was this new virus news coming in from China – a lot of uncertainty. We were putting out a lot of fires. There was some product roadmap, I don't exactly remember what, but two weeks into my job at OLX, we entered the German lockdowns. I think other countries had rolling lockdowns around that time. But Germany had locked down two weeks after I joined and there was a lot of uncertainty. There was a focus from management to really work on only essential stuff and things that will keep the business alive, because back then we didn't know how long this would last, what the financial implications would be and stuff like that. Basically, a lot of our moonshot projects were put on hold. (10:37)

Sadat: I had the time I needed to look at the landscape and understand what our main problems were. What I had identified as one of our main problems was the fact that we were still very much coupled to the big monolith. This posed some challenges plus the configurations with our Solr cluster were suboptimal. I created a proposal, shared that with my manager and my product manager – to decouple search. It was an interesting idea. My ask to them was “Give me six months with zero product feature development, but core engineering work, and I will promise to give you the fastest iterating product at OLX.” Three years later, I think, the relevance team, which is now my feature team, is one of the fastest iterating teams at OLX. Last year, I think we did over 100 experiments with new product features. (10:37)

Sadat: My management, my leaders, believed in me, they gave me the opportunity, and humbly I think I achieved a lot of what I had promised. There were some delays – I think it took us a little more than six months. But we eventually got there. We decoupled search, we improved the configurations, and we also have backups upon backups now. So I guess the last time we had a full outage on search – it's probably been over a year or two years. (10:37)

Sadat’s pivot point as a backend engineer

Alexey: Maybe I'll try to summarize what you said. You and your team had a super large service that was doing everything, including search. Search was a little piece of this. Making changes to this little piece was very difficult because it affected the rest of the service. You approached your management and suggested, “Give me six months, I'll take this part out into a separate service and it will be a micro service. We'll be responsible only for search.” (13:07)

Sadat: Macro service is what I like to use. Macro is a little more than micro. I mean, it is a very broad service – it does a lot of stuff. But in essence, yes. We wanted to carve out a bit of the Big OLX Monolith. This helped us with a lot of stuff. The stack was newer – alerting and monitoring were thought of before the product development, so it was very natural to have alerting and monitoring in place. We had a lot of experience running the old system, so we knew what systems needed to be optimized or what parts of the code needed to be optimized, what parts of the code could be rewritten in different ways. In essence, that's what we did. (13:40)

Alexey: What did you use for that? For implementing this thing? (14:26)

Sadat: The new service, the service that is currently live, was written in Kotlin. There are a few other satellites around the macro service, which are written in Python and Kotlin, depending on what the services are. Search at OLX also consumes a lot of machine learning products: spellchecker, RANCOR – all of that. So we do have some Python services that are in the ecosystem. But the main chunk is Kotlin. (14:30)

Sadat’s exposure to ML and Data Science

Alexey: What was your exposure to all this machine learning and data science? I guess before OLX, or at OLX, how much of this stuff did you already know? (15:03)

Sadat: My first exposure to machine learning, artificial intelligence, was during my Master’s thesis. The thesis that I published was basically to propose an algorithm that was using a neural network to predict the energy consumptions at the Stuttgart Airport. It was in collaboration with a company that is responsible for maintaining and running the Stuttgart airport. That was my first exposure – that's when I really got into the details of what neural networks were and how they functioned. I built my first network then. Transfer learning was definitely not a thing back in the day, so I created my own network, played around with deep nets, shallow nets – all of that stuff – but only for the purposes of time series forecasting. (15:13)

Sadat: So that was my first exposure. Then I was doing search at reBuy. This is my first company in Berlin. A couple of years into my job… reBuy is an amazing company, by the way. A plug to them, not that they need it – they offer engineers 20% of the week to work on pet projects. The only condition is that these pet projects should not be completely ludicrous. If you were to propose you want to learn knitting, they might say no. But for any tech-related projects, they're super game. A colleague of mine, Machu, (love the guy, by the way) I don't know where he's based these days, to be honest but he's still at ReBuy. Not in Berlin anymore. Machu and I decided to work on spellchecker, because that was a problem that we were facing at search/product at reBuy. (15:13)

Sadat: We wanted to use the 20% time to try and implement a deep learning spellchecker. Long story short, we did not succeed, but we learned a lot more about machine learning and about training algorithms. That also actually kickstarted the inspiration to then start the Search Tech Community in Berlin, which is a community where we host meetups – shameless plug to that as well. [chuckles] I hope it's not uncommon for people to plug things and the various activities that they're associated with. (15:13)

Alexey: Definitely not. That’s why people come to the podcast, I guess. [laughs] So you started this meetup (this community) while at reBuy? It was a while ago, right? (17:37)

Sadat: It was a while ago, yes. I think the community is over four years old now. No, it's five, six years old. It's an old community. We actually went on a sabbatical during most of the corona pandemic – for two years. That's why it feels much shorter than it actually is. It’s like two years of my life I've been erased. I forget to factor that in. So that's when the community started. It got me connected to a lot of other people working on search, working on recommendations, because usually, ranking is a very common factor in both of these things. (17:47)

Sadat: Then somewhere during this time, in this community, both Machu and I were inspired to give up the spell checker and pick up recommendations as a project – as a problem that we wanted to solve. We trained a few algorithms. In the end, word2vec was very popular in those days. A lot of teams were doing it. I think what inspired us was probably Pinterest pin2vec – there's a lot of these “2vecs” out there. [cross-talk] (17:47)

Alexey: Did you actually find a paper? Or was it more like a video? (18:55)

Sadat: No. We actually tried, to be honest. Machu I think is more disciplined than I am in this. We did find a few official papers on the word2vec architecture. But then, I think in the end, it was videos. For me, it was mainly going through videos and actually hacking code. I gave up reading and just started doing stuff. I remember the word2vec model because that's the one that succeeded in the end. A year from there, I think what ended up happening was that the search team became the “search and recommendation” team. Before this, reBuy was buying recommendations from a SaaS provider and our model was actually better. We did some A/B tests and we were 2-3% better in transactions from recommendations, compared to our competition. I think we ended up saving half a million euros or something like that for the company, with a model that we had trained in the 20% of our time. (18:58)

Sadat: That was when I was very much involved. The recommendation project was something where we worked on the training, the data engineering, the data gathering – and all of the nitty gritties of probably what most data scientists do today, all the way up to taking the model to production, hosting it on our AWS account, and then having a job that would retrain it. I had very little professional experience in machine learning back then. Now that I've worked with some amazing practitioners and machine learning engineers, I laugh at what I had pushed to production. But it was not very different from what the experts are doing today. (18:58)

Alexey: Just curious, how much of the theory did you already know and how much did you need to know to actually do all of that? Or was it completely hacking? (20:37)

Sadat’s Act Before you Think approach (with safety nets)

Sadat: It was a lot of… the internet. I love the internet. You don't need to know anything. You just copy/paste and brute force till it works. That's one way that I work. And it does work. To be honest, I understood the word2vec model after I had implemented my model. I mean, I conceptually understood why it works and what it does after I saw it work for the recommendations. Before that, for me, it was just, “Yeah, something that works. There are a bunch of companies that are using it. Recommendations is one use case. Let's see what we can do.” (20:47)

Alexey: I think that’s pretty typical for engineers. This is how I learned this thing, too. Just try it and see what happens. (21:21)

Sadat: “Act before you think.” [Alexey agrees] It's not good advice, but for engineering (and if nobody's life depends upon it) it's not bad advice. (21:31)

Alexey: What can possibly go wrong if you're just on your laptop? It’s one thing if it's in a production system and the entire search service goes down. But if it's just your laptop, and you’re checking if this thing is working or not, then it's another thing. Right? (21:40)

Sadat: Absolutely. One thing that I've always told colleagues and also engineers who’ve worked with me is “Invest in safety nets, because once you have a strong safety net, you're not going to be afraid to try stuff. Feature flags, experimentation, backups, alerts, monitoring, etc. If you have a robust system that is designed in a way that failure is not, “if” but “when”. So it's never a question of if you'd fail, it's a question of when you will fail. Prepare for that. If you have that done, you can be very bold and make maverick decisions. That's how you're going to succeed. Because to be honest, it's a very fast-paced world we live in. We can't be cautious all the time. (21:58)

Alexey: In summary, just pick a project that might benefit from machine learning, implement it, make sure there are safety nets – if anything goes wrong, there is a switch that you can just take it down or just turn it off. That's basically it. Right? (22:46)

Sadat: That's your playground. Once you've got your playground setup – play. That's the only way you will learn to be good at the game. You have to practice. Another thing I tell myself is “I'm not a heart surgeon. Nobody's going to die.” [chuckles] (23:03)

Alexey: Well, maybe there is a user who really wants to find a particular pair of shoes and they can't because the search is down. (23:19)

Sadat: It's a few euros or dollars lost. It's okay. But again, we should be bold. Don't be afraid of making mistakes. You learn more from your mistakes than you will from otherwise. (23:25)

Sadat’s street cred and transition into management

Alexey: Okay. Coming back to your OLX days – you saw that there was a problem and you wanted to solve it. You were new then, right? You just joined. You came to the management and said, “Okay, give me six months. I'll make some magic and then it will work really fast.” (23:39)

Sadat: I did not promise magic. [chuckles] I had an architecture diagram that I had proposed. [Alexey agrees] To be honest, it may sound cooler than it is. Like I said, before OLX, I had done search at Delivery Hero. And before that I had done search at reBuy. Unsurprisingly… [cross-talk] (23:56)

Alexey: So you had some street cred, right? (24:15)

Sadat: Not only did I have street cred, I had done the exact same thing in those two jobs. That's how the industry evolves. (24:18)

Alexey: That's why you were hired, right? (24:27)

Sadat: I guess. When companies are hiring, when we are hiring, when I am hiring – if you have prior experience with what I'm planning on doing, you obviously have a better chance. I was invited to the interview because of the meetup that I would host. People who had invited me and knew me from before my interviews – it helps to be present in the community and to, like you said, have street credit. It definitely does help. (24:29)

Alexey: So management trusted you with working on that thing. You spent half a year, it worked, and then you got promoted to an engineering manager. Was it like that? (25:01)

Sadat: No. No, it wasn't like that. [both laugh] But this definitely was a very strong point in making it happen when it happened. After we had scaled out, six or seven months later, we were also more certain about how the world was evolving. At this point, six months into the pandemic, online businesses were thriving. There was a lot of business happening, money was flowing in, and so companies started to reinvest in themselves. (25:11)

Sadat: The search team was growing. There was a little bit of a reorg and it became the “relevance team” as it is now in its current incarnation. The idea was to deliver customer value. We were doing experiment after experiment, A/B test after A/B test, prod features were getting launched. It definitely did add to my credit, that I did advocate for something that was not very obvious, but it definitely helped us move forward. (25:11)

Alexey: What was that? (26:27)

Sadat: The decoupling project – moving out of search. To be honest, the way I became an EM was that my existing EM decided to leave the company. That created the opportunity. You need to have the capability, but you also need the opportunity – and you need luck to have them both. I think at this time, I had that around. My manager decided to leave OLX. I had been at OLX at this point for a year and a half or so – actually a little more than a year. And then things started falling in place. (26:28)

Sadat: I interviewed. The way we do it at OLX is, if an IC wants to move. I think after a senior engineer, especially a senior engineer moving into an engineering manager's position, it's a serious recruitment process. I went through the entire process, including the technical screening – the recruiter screen, which, in my opinion, seemed a little superfluous, but… [cross-talk] (26:28)

The hiring process as an internal candidate

Alexey: It was as if they were hiring somebody external – completely, the entire process. (27:40)

Sadat: It was completely as if I was applying for the first time. Well, I think, that's also the intention. OLX wants to treat you as an external hire because at the end of the day, we want the best person for the job. There are certain benefits to being an internal candidate. You can showcase what you've done in the past and they will believe you 100% because they see it. Whereas, if you're an external candidate and you say you've done something but you can't prove it, or your story doesn't add up – that could raise suspicion. (27:45)

Sadat: Also, as an internal candidate, I could ask for people I've worked with in the past to maybe give me a recommendation or to put in a good word. Those are things that maybe some external candidates might not have. But if you have good connections in the industry, that's also doable. So I did the interview, went through the entire process and only then was I offered the position of the engineering manager for search and recommendation. (27:45)

Alexey: Do you remember the questions? (28:47)

Sadat: It wasn't related to search, if you're asking that. [chuckles] (28:51)

Alexey: What was it related to? (28:55)

Sadat: A lot of my interview was related to people management. Maybe a little anecdote – before I applied for the EM for search and recommendations, there was another internal opening at OLX in a different part of the company for an EM position, and I applied there as well. I was actually rejected. But after I got rejected, I spoke to the hiring manager, because we're all OLX employees and he gave me an amazing breakdown of why I failed. What he said was “You didn't really fail, but you probably need a little more to pass the bar.” (28:57)

Alexey: That’s something that external candidates usually don't get. Right? (29:31)

Sadat: Exactly. They won't get such detailed feedback. Usually, if you're an external candidate, your feedback is delivered via the talent acquisition team. In this case, I was directly talking to the director of engineering who was the hiring manager. He gave me very good points. I thank him for that even today. Dian, if you're listening (he’s at Flink now, I think, with their engineering team) amazing feedback. I used a lot of that in the second round. And this time it was a success. (29:35)

The importance of people management skills

Alexey: What kind of people management questions can people expect? Maybe don't give away the exact questions you were asked, because maybe we still use these questions. [chuckles] But just to get an idea. Is it about conflict resolution? Is it about promoting people, firing people? Or what? (30:09)

Sadat: To be honest, the questions aren't that important. What the hiring manager is trying to understand is, “Are you a person who is capable of leading the team that is in question?” Usually, an engineering manager or managerial positions, at least at OLX, are very specific to teams. That's what they're trying to understand – if you have a team that is riddled with conflict, they will be looking out for a person who is better at conflict resolution. If you are trying to lead a team that is very technical and developing a platform type of solution, they’d look for a person who's more technical than otherwise. The advantage that I had with my interview for the search team was that I would safely say I am well-versed with the field of search and recommendation. So I had that to my advantage and I had worked with this team before – I have been working with this team. That was also an advantage to me. I guess these weren't advantages in the other interview that I had failed. (30:25)

Sadat: At the end of the day, the hiring manager wants to find a person who is capable of leadership, rallying the team behind in tough times, celebrating with the team in good times and is generally somebody that the team looks up to, respects, and is happy to work with. From a business perspective, the engineering manager's job is to keep a healthy team, add business value, make sure that attrition is controlled – hiring new engineers is very expensive, hiring replacements even more so. Those are some business metrics that need to be optimized and then any person who can fit that profile is the person for the job. (30:25)

Alexey: But in terms of maybe any actionable advice? Suppose I want to interview now for an engineering manager position. What can I expect? How can I prove all these things that you said? How can I prove that I can do them? (32:11)

Sadat: Usually, what also works is having… it's funny, like the chicken and the egg problem. When you're in an interview, they'll ask you for evidence that you have, for example, led a team in a situation or had experience in resolving conflict inside a team. If you're interested or you're planning to appear for such an interview, I would really ask you to sit back and think about the instances in your career where you have shown leadership – it could be a small project that you've led. (32:28)

Sadat: It does not need to be a group of 10 people, it could be you and a few other mates – a college project that you did, where you acted as a leader. So showcase that ability. Conflict resolution and business acumen is something that is also important when it comes to these positions. So you need to understand things like this. Look back at your career, see instances where you have shown this, and highlight those. (32:28)

The Brag List

Alexey: So let’s say there was a conflict in one of the projects that I had, then I should take note of it, and understand what my actions were – how I helped to solve this conflict – and when there is a question about that, I bring this up. Right? That's the kind of homework I need to do? (33:31)

Sadat: That's the kind of homework. Right. Ilias, my old manager who had to leave OLX for me to get his position, always asked me to maintain a brag list. Maintain your brag list. (33:46)

Alexey: What is that? (34:00)

Sadat: Whenever you do something worthy of mention, note it down – because oftentimes you forget that you've been an awesome person before. When it comes to such interviews, it's very hard to lie. The people who are interviewing are experts, so you can't possibly lie to someone and get away with it. The only way is to actually tell the truth. And in order to tell the truth, you need to remember that you've been in a situation like this. Generally, in a 30-minute or 45-minute interview, you cannot go through your entire life. So it's better to do that before – create your brag list and then walk into the interview with your brag list. (34:01)

Alexey: Did you actually have it open during the interview? (34:39)

Sadat: Did I have a brag list? Or…? (34:42)

Alexey: No, did you have it open on your screen when you had the interview? Or you reviewed it before the interview? (34:45)

Sadat: I reviewed it before. It was a Sublime text document – very chaotic, like most things I do. If I had it open during the interview, I would have crashed. (34:51)

Alexey: Did you put it on Dropbox? How do you keep it? (34:58)

Sadat: No, no. My method of organizing files is never closing a Sublime text document. (35:01)

Alexey: [thumbs up][laughs] Good. Your entire brag list was there? (35:10)

Sadat: It's still there, to be honest. If I spin it up, I have like 300 tabs in my Sublime text, which are basically documents that are not final enough to be saved, but not untidy enough to be forgotten. And the brag list is one of those documents. (35:13)

Alexey: Do you recommend this approach? [chuckles] (35:28)

Sadat: No, I absolutely do not recommend this approach. [chuckles] What I've become better at now is, like I said in the beginning, I use G Suite a lot. Now I organize my life around Google Docs. It's one of the best tools that I have. (35:30)

Alexey: Yeah, my favorite too. [chuckles] (35:47)

Sadat: Yeah. So I'm gradually moving away from Sublime. I wouldn't advise that to anybody. (35:49)

The most difficult part of transitioning to management

Alexey: What was the most difficult part of your transition? (35:57)

Sadat: I don't know if a lot of ICs or engineers realize this or not, but we have a lot of dopamine in our daily life. [cross-talk] (36:02)

Alexey: You mean when you code, right? (36:14)

Sadat: When you’re coding, right. You create a merge request – bam, that's dopamine. You get an approval – another shot of dopamine. You hit the merge button – dopamine. Deploy – dopamine. A/B test started – dopamine. There’s dopamine throughout the day. It is a very addictive career. And that's why people enjoy it, right? That's why people who are good at it get very successful at it. People spend time even after work. It's very interesting and especially challenging for people who are into it. As a manager, those dopamine shorts stop, so withdrawal is what you go through, really. (36:16)

Sadat: That is the analogy I really want to use – like a drug addict who stopped getting his drugs. I was very close with my previous engineering manager. We spoke a lot about what would happen during the transition. He prepared me for taking on the team. There's a lot of prep work that goes into it. Some people are lucky enough to have a mentor like I had. And some just figure it out. You learn how to ride by riding a bicycle. But nothing prepared me for what I actually felt. The first few weeks, when you have nothing – the thing is, the time it takes to see impact stretches from hours or minutes to weeks, months, maybe – a quarter sometimes. (36:16)

Alexey: Because the impact is not done by you. Right? [Sadat agrees] You need to coordinate this impact – you need to make it happen, but not do it yourself. (37:48)

Sadat: But not yourself. And that's a very key component. What a lot of early managers do (I think, I've done this) is when you're not getting that dopamine, you immediately lapse and you go back to becoming an IC – and you start coding again. Because there is your shot of dopamine – you feel that hit, like “Now I'm useful. I'm doing something again.” I made that same mistake. I went back to coding because that was easy for me. But then at some point, I realized that every minute that I am sitting with my IDE writing code is a minute that I am not doing something more important for my entire team – which could be talking to another stakeholder somewhere, getting their buy-in for a crazy project that we want to do, talking to a data science manager to get his team to implement a data science model. (37:57)

Sadat: Every minute that I was with my IDE, was a minute I was not doing those things. But somebody has to do those things, so they would start stacking up. And then you're just sitting at your computer at 8 PM writing slack messages to people – your day becomes crazy. You have to stop being an IC and start focusing on what needs to be done. (37:57)

Alexey: So how do you get your dopamine dose then? Chocolate? [chuckles] (39:04)

Sadat: [chuckles] I do have some chocolate bars on my desk right now, but no that's not how you should be doing it. Otherwise you will put on a lot of weight. So one is – be prepared. No matter how prepared you think you are, understand that you're not prepared. What you're losing out is quite a bit. Trust in the people that you've talked with about this transition in the past – your mentors, other people from the community, from the industry – they know what they're talking about. This is a longer game. I don't play golf, but I imagine this is how golf would seem like. You just keep going behind the ball, shot after shot, and then three hours later, bam – somebody's a winner. I don't know how that game works. (39:11)

Alexey: Like chess, maybe, as well? (39:57)

Sadat: Yes, maybe. But in chess, the minute you take down the opponent's thing, there's a bit of dopamine there. [Alexey agrees] Chess can be a very-fast paced game. Think of the most boring game and then that is your analogies. (39:59)

Alexey: It’s golf, right? [chuckles] (41:14)

Sadat: [chuckles] Golf is what comes to my mind. And you need to be very strategic. What you need to do is, at least what worked for me, pick up a project and make that project your source of dopamine – or the completion of this project. And then drive it – put all your energy into that. A month or a quarter later, when you've delivered it… (40:16)

Focusing on projects and setting milestones

Alexey: By “project” you mean that you work on as an IC or that you have other people work on? (40:41)

Sadat: No. By “project” I mean something that is now on your new desk. This could be something that you were working on in the past or not necessarily – but something that needs to be done for the team now. Pick that up, talk to your engineers or to whoever is on your team, talk to stakeholders – invest yourself in completing this. Focus on one project (at least that works for me) laser focus on one project and you will learn a lot from the mistakes that you will make during the execution of this project. (40:48)

Sadat: But when you have reached the end, and then when you get that one big shot of dopamine, and if you're cut out for it, because I think a lot of people just aren't cut out for the job. If you're cut out for it, you'll know you've made the right decision. And if you're not, you'll know you've made the wrong decision. And then you will gradually try to go back to where you were before. (40:48)

Alexey: Tell us more about this project. I don't think I understood. So this is a project that you’re working on as an actual developer – you actually go there…? Or you have somebody? (41:45)

Sadat: No, no, no, no, sorry. Somebody works on it. What I meant was a project that needs to be done – that needs to be executed in your team, you're leading this team. (41:55)

Alexey: So you’re coordinating this project. (42:08)

Sadat: Exactly. Your team has to execute this project and you're the leader. They're behind you, they're supporting you, but you have to bring it home. (42:09)

Alexey: But what if you're craving a little bit of dopamine before that? Because that will happen in three months, but you want to have dopamine now. (42:18)

Sadat: Milestones, that's what I do now. (42:26)

Alexey: Milestones. Okay. [chuckles] (42:28)

Sadat: Set milestones. When you have clear milestones, and if you're doing it right, your milestone should not be a quarter ahead. It should be more like a sprint ahead. Those can work. But I think for early on in your management career, it might be very hard to get all of this right. I've made my mistakes, and the people who are listening are going to make their own mistakes. (42:30)

Sadat: But in the end, I think what I want you to take home is – understand that it is a completely different reward structure. Your reward is going to be much bigger and much more satisfying, if you're into it, but it will come at a delayed time. (42:30)

Sadat’s transition from EM to data science management

Alexey: We also wanted to talk about what you do now, which is not an engineering manager anymore, but data science management. So tell us about that. Maybe it was something similar? You worked as an engineering manager for some time, and then you applied for a data science manager position. Right? (43:12)

Sadat: Again, I went through the interview process. The way we hire data scientists at OLX is very interesting. I love the process, by the way. We have a panel interview, which we don't have for engineering. So I had to go through the panel interview. I did not have to go through the recruiter screen this time. Thank you, recruiting team, for that. But I did go to the panel interview, I had to present my case, I was given a case study, and asked on how I would execute it. (43:33)

Sadat: I spoke with my team – my future team, the one I would be taking up and I spoke with the manager who was leading the team. I spoke with leaders inside data science. I also had a lot of anxiety. I wasn't sure. I wanted to do this – I desperately wanted to do the transition, because I had always been on the margins with data science. I had played with models, but never owned them. I deployed models but never really controlled them. I wanted in on the fun. (43:33)

Alexey: So that's why you decided to apply for a data science manager position? (44:33)

Sadat: The way I see my work is – there were challenges that I had as an engineering manager that were fun, but in data science, I feel like I'm enjoying the challenges more because the problems are less defined. It's a little harder to understand what really needs to be done. Building solutions is not as straightforward. The dopamine is even further delayed now, but it's even bigger chunks. For example, there's a topic that I have on my plate that I could maybe share. It’s regarding chat and moderation. (44:38)

Sadat: Currently, I'm leading a team of data scientists who are part of the trust and safety pillar at OLX. What we do there is – maybe just a little bit about that, before I get into some of the challenges that we face. OLX – as most of you viewers know by now, because you've been at this company for a while – is a platform which facilitates user-to-user sales. All of the content is user-generated and you can imagine that people are creative – they always want to game the system, or fool the system, or abuse the system. Our users are no different. And we have a very strong team of people in operations (moderators) in product engineering and data, that are building solutions to make sure that the content that is against our community policies, is not allowed to go live. This includes, for example, even chat messages. You can imagine people trying to abuse each other in chat, or scam other unsuspecting users. Preventing this is part of my team's job. But that is the problem. (44:38)

Sadat: People want to chat with each other – they are good people who want to chat with each other and they are bad people. Identify the bad actors, hide the messages, whatever – stop it from happening. This is somewhat different than the challenges that I had in my engineering team, which were a little more defined. Now I get to work with data scientists and discuss machine learning solutions, in this case, a lot of NLP. I need to understand the context of not just a message, but the entire chat, like, “What is really happening here? Is this a genuine conversation? Or is this somehow fraudulent? Is one person abusing the other? Are community guidelines being violated here?” Our idea is to develop a solution that would comprehensively understand what human communication is about, and then make a decision. Versus the engineering approach, which is “I'm going to put a filter for the word F-U-C-K and if somebody types it, I will block it.” (44:38)

How much domain knowledge is needed for management?

Alexey: How much do you think a data science manager should know about all these things? NLP is a pretty large topic. [Sadat suggests “nice”] I don't know about “niche”. There are a lot of things happening there. Maybe coming from a software engineering background, you don't necessarily know it in depth? So how much of this do you need to know in order to be helpful to your team? Do you even need to know that? Or, as a manager, you help differently – not by pointing to papers? (47:50)

Sadat: Right. I don't point people to papers. They point me to papers. I think there are different types of manager archetypes. You will always have the manager who's a domain expert and is going to lead by being the domain expert. I am not a domain expert, neither in the topics of trust and safety, nor am I an expert in the topics of machine learning or data science. My focus is genuinely around coordination. (48:21)

Sadat: I do understand machine learning and data science to a degree where I can have an informed discussion with product and tell them, “Hey, this is a problem we can attempt to solve,” versus “This is a problem that is better solved with an engineering solution,” or “Better solved with a human-in-the-loop solution.” (48:21)

Sadat: But from then on, my job is to filter the right kind of problems, set the right kind of success metrics, and share that with my team. Then they take it from there. They propose the solutions and they propose the models. I am part of the discussions, because I'm genuinely curious as to how they function, how they propose these things, but I am not a contributor to a great deal in any of these discussions. (48:21)

Sadat: To answer the question “How much do you need to know?” I think it's definitely helpful to know if the problem that is being discussed will have a successful machine learning solution or not. That definitely helps you. But beyond that, it's not necessary. If you know, I think it's going to be fun, but it's not absolutely necessary. (48:21)

Alexey: Then, I guess, for mentoring the team, you rely on more senior team members? (49:48)

Sadat: Yes. The way structure at OLX – you, yourself, are a principal and I have spoken to you on occasion to help people in my team upskill with technologies. That's also coordination, right? I know that there are people who are better at this than I am. Why should I be... I just connect experts to people and that's it. I facilitate discussions, but I don't drive or own them. (49:54)

The main difference between engineering and management

Alexey: How different is your job compared to what you were doing previously? Now, I guess you have more interaction with stakeholders and the nature of this interaction is slightly different, because you need to keep in mind all these things that you mentioned, like, “Does it need a machine learning solution? Or can we just go with a pure software engineering solution (i.e. filters)?” Things like that? Is this the main difference? (50:21)

Sadat: I think this is a significant part of the difference, yes. I am way more involved in product discussions than I was previously. But I also think that a part of this is how your company looks at the different profiles. I think at OLX, a data science manager is really a counterpart to product and engineering. And my job, the way I see it, is to work with the others from the product and technology side, to propose better solutions – to help them make decisions and, in the end, drive customer value and business value. My discussions are far less technical now and far more impactful than they were previously. (50:44)

Alexey: Do you think you could have just jumped one of the roles? [cross-talk] Just keep the EM part. (51:34)

Sadat: Here's my honest take. Like I said, you will always have to prove yourself before somebody gives you the keys. How does a software engineer prove that he can be a good data science manager? I'm not saying it's impossible, but I think it’s… [cross-talk] (51:43)

Alexey: Well, you did a fair amount of ML-related stuff as a backend developer. (51:58)

Sadat: Could I have done it? I think… [cross-talk] (52:04)

Alexey: Or were you just helping? (52:07)

Sadat: We'd have to ask the hiring manager if they would have taken me. [chuckles] What I do know is that during my interview for my current role as a data science manager, I did use a lot of my success stories from my previous role, which was an engineering manager role, to take on this position. Had those success stories not existed, would I have still been given the opportunity? That’s something with the hiring manager and the committee that interviews would answer. Like I said, we had a panel interview – so the panelists. (52:10)

How being an EM helped Sadat transition no DS management

Alexey: But as an engineering manager in the search and recommendations team, you got involved in all these machine learning-related topics, which gave you some credibility, right? (52:40)

Sadat: Definitely. One of the benefits of being an engineering manager for the search and recommendations team, for me personally, was this is one team that works very closely with data science. The relevance team is a big contributor to the success of the data science work. Not only did it give me the platform, but it also allowed me to showcase my working style, who I am, and what I do, to my future employers. That definitely added value. (52:52)

Alexey: The future employers are here, at the same company, right? (53:23)

Sadat: Yes. [chuckles] The same company, different hiring manager, if you will. (53:26)

Transitioning to DS management from other roles

Alexey: We have a few questions. One of the questions is… I don't know if you can answer this, but maybe you saw some of your colleagues do this. The question is, “What are the challenges of getting a data science managerial position from other roles, for example, from data engineering or analysis?” I think you were doing something similar to data engineering, as I understand? [Sadat agrees] We actually talked about that, more or less. (53:32)

Sadat: Right. I guess it shouldn't be difficult, depending on what your hiring manager or the potential team that your company that you're applying to – what they're looking for. I would say, with that experience – so if you have experience as a data engineer or a data analyst – I think you know a fair deal of how the data science teams work. You also understand a lot of the stakeholders, so that is to your advantage. You need to focus on your leadership skills. Why are you the right person to lead this team? And if you can convince them of that, then I think you should get the job. (53:56)

Alexey: That's the homework we talked about, when we were talking about getting an engineering management position. (54:27)

Sadat: Yes. Create your brag list. (54:35)

Alexey: All that advice you shared with us about this EM position also applies to data science. Right? (54:38)

Sadat: Yeah, exactly. Definitely applies. (54:46)

Alexey: Okay – but keep this brag list in a Google doc, right? Not Sublime? [chuckles] (54:49)

Sadat: Keep it in a Google document – not in an unsaved Sublime text document. [chuckles] (54:53)

Alexey: Why did you not save it? (54:58)

Sadat: Every time I tried to save it, a file name… I don't know. No reason. I'm thinking about it, and there's no reason. I could have just called it braglist.txt and saved it. I didn't want to call it a brag list. Maybe a little bit of that. The name is… [cross-talk] (55:01)

Alexey: Call just “a list”? (55:20)

Sadat: Yeah, maybe. It's an outrageous name “brag list”. (55:22)

Alexey: Or “work list”? (55:26)

How to feel accomplished as a manager

Sadat: Accomplishments. That's a better name. It's still called “brag list” on my Google Docs. (55:26)

Alexey: So you still keep it? (55:37)

Sadat: I still have it. I still add to it whenever I feel that I have done something worth mentioning. (55:39)

Alexey: Yeah. How do you… because you actually didn't do it, right? Somebody else did this. You were just standing there. [chuckles] At least, when I was managing a team, this is how I felt. (55:47)

Sadat: What you need to understand – this is a very good question – yes and no. Because the thing is, had you not been there, do you think similar outcomes would have been achieved? (56:00)

Alexey: How do you know? You cannot test this. [laughs] (56:11)

Sadat: You cannot test it with an A/B test, but you can, for example, go through the journey of the team or the project, and see areas where you had key insight, and where what you uncovered or what you discussed changed the direction of the project. This is going to be there. If you are doing your job right, you are going to find instances where, had you not been present the outcome would have been slightly or significantly different. That is what differentiates a good manager from a bad one. At the end of the day, if you're getting paid – and I believe this firmly – if I am getting paid to do something, the value that I bring should be more than the value that I take out. It's simple math. (56:16)

Alexey: That's the validation, right? If you check your bank account and you see some money, then you're still doing well, right? [chuckles] (57:07)

Sadat: No, I would want to evaluate it even closer. [Alexey chuckles] Not only on that high. That is definitely a very good evaluation, but I would also want to have other “health metrics,” so to say. (57:15)

Alexey: What are those metrics? (57:30)

Sadat: For example, there’s one thing that I do look out for – this isn’t very concrete. What I do look out for, for example, now as a data science manager, I have a lot of interactions with stakeholders and product managers, and engineering managers – and the amount of influence that I have, where I can change their opinion from one to another and then, “Does my proposed solution deliver value?” If A and B are there, then I am very happy at consuming the money that my employer deposited in my bank account. If not, I would want to get feedback and improve. (57:34)

Alexey: Okay. And you put these things in your brag list. (58:13)

Sadat: Some of it. When it's successful, it ends up in the brag list. But when it's in progress, there's probably another unsaved Sublime document somewhere. [chuckles] (58:17)

Sadat’s book recommendations

Alexey: [laughs] Okay. Do you have any books or other resources in mind that you can recommend to the listeners? (58:28)

Sadat: I have read a few books on management. I can recommend a few books on management. But my style has always been “Ride a bicycle to learn how to ride a bicycle.” You have to do it before in order to understand how it's done. It's a very agile methodology, “Start, and then iterate and improve.” One book that I read is called The Manager's Path. It's written by a Facebook employee (or an ex-Facebook employee) and she describes her journey from the early days of Facebook and how it scales. I think that that will definitely help. There are a few other books that don't stand out because all of the stuff that you read in them is stuff like you hear in a podcast like this, right? Somebody else's journey – it might or might not relate to you. Make your own journey. (58:36)

Alexey: I saw this book, actually, at OLX. (59:25)

Sadat: It's quite a recommended book. (59:28)

Alexey: It’s on the desks of quite a few people, especially those who just became engineering managers. I saw that they have this book on their desks. It was before Corona, when people actually came to the office. (59:30)

Sadat: There's another one, from the founders of Netflix, No Rules Rules: Netflix and the Culture of Reinvention. It's more about culture, how you want to drive your team's culture. I think that's a very important part of being a team leader. I like that book. It's unique. We are definitely not running OLX with those principles. But it's nice to see how certain rules are maybe superfluous and can be done away with. The story is – make your own journey. Understand and learn from others, but make your own. (59:44)

Sadat’s meetups

Alexey: When is the next Search Meetup meeting? (1:00:20)

Sadat: Oh! So glad you asked. I'm not hosting this one under the banner of the Search Meetup, but I am going to be hosting another meetup at OLX on the 7th of December, 6 PM. The topics are going to be around bias and fairness in AI. How would you measure that? And how would you counter that? It's co-organized by wonderful people at Toloka, which is kind of like Amazon Turk – human annotation crowdsourcing. So if you're in Berlin and if you want to enjoy good food, good conversation, and networking with great people – then show up at our offices at Alexanderplatz. [Alexey agrees] You might also see Alexey there. (1:00:22)

Alexey: I don't promise that. [chuckles] But you will definitely see Sadat. [Sadat agrees] Okay. Well, thanks a lot, Sadat. It was a pleasure talking to you. Thanks for sharing all these stories, your experience. That was fun. Thanks, everyone, for being here. (1:01:12)

Sadat: Thank you. This is a pleasure. Have a nice day. (1:01:33)

Subscribe to our weekly newsletter and join our Slack.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.

DataTalks.Club. Hosted on GitHub Pages. We use cookies.