From Theme Parks to Tesla: Building Data Products That Work | Abouzar Abbaspour

The transcripts are edited for clarity, sometimes with AI. If you notice any incorrect information, let us know.

Career journey and early motivations

Alexey: Hi everyone, welcome to our event. This event is brought to you by Data Docs Club which is a community of people who love data. We have weekly events and today is one of such events. This is a bit unusual. We usually have them on Mondays. (0.0)

Alexey: But actually this week we have quite a few of these different events. I hope you are enjoying it. If you want to find out more about the events we have, there is a link in the description. Click on that link and check it out. You will see all the events we have in our pipeline. (12.0)

Alexey: Then do not forget to subscribe to our YouTube channel. This way you will get up to date with all the streams of future interviews that we will have. And last but not least, do not forget to join our Slack community where you can hang out with other data enthusiasts. (30.0)

Alexey: During today’s interview, you can ask any question you want. There is a pinned link in the live chat. Click on that link, ask your questions, and we will be covering these during the interview. Okay, that is the mandatory slides that we have before each interview. (41.0)

Alexey: Now I am opening the questions we prepared for you. And Abouzar, if you are ready, we can start. (1:05)

Abouzar: Yeah, let’s do it. (1:11)

Alexey: Today on the podcast we are joined by Abouzar. He has worked on data and ML projects in very different places from amusement parks to e-commerce and now Tesla. We will talk about how to turn data into real products, what it means to productionize machine learning, and what is the future of data and ML engineering. So, welcome to our event. (1:17)

Abouzar: Thank you. Happy to be here. (1:35)

Alexey: So, let’s start with your career journey so far. Can you tell us about this? (1:41)

Abouzar: Yeah, so my background is software engineering. I got my bachelor and master in software engineering, basically computer engineering, from Iran. Then I worked for a couple of years in a telecom company. After a while I thought, okay, this is getting boring. (1:48)

Abouzar: Since you have to develop a lot of things and also support them, after a couple of years it becomes a lot of things to support. So I thought maybe it is time to go after something I like more. My master thesis was also on this same topic. It was machine learning, more like data science topics. (2:06)

Abouzar: So I was just looking for different universities that had a similar program, where I could further study data science and understand it more. I came across a program in Eindhoven University that was called Professional Doctorate in Data Science. Now it is called Engineering Doctorate, they changed the name because of overlap issues. Yeah, I applied, when was it… (2:35)

Alexey: Sorry, when was it? Was this 2018? (3:13)

Abouzar: 2018, yeah. (3:19)

Alexey: I was applying a little earlier, as we spoke before the interview. I was also considering Eindhoven University for my master’s, and I think the beginning of my career journey was quite similar. I was a software engineer, and I was getting bored, working at a bank, and I thought, do I really want to do this for the rest of my life? (3:25)

Alexey: Then I found out about machine learning, and I thought, oh, that is so cool. Eindhoven University was one of the universities I considered applying to. It is very interesting. I do not remember what it was called though, but I remember that university, and I really liked the program. In the end, I chose a different one. (3:39)

Abouzar: Okay. Which university did you go to? (4:09)

Alexey: So I graduated from TU Berlin. Oh, okay. And now I am in Berlin. (4:14)

Abouzar: Nice, nice, nice. Yeah. So those years, I guess, it was like OpenAI just launched their thing. They started maybe two or three years before, and it was a hot topic, especially in the Netherlands and also in Germany. You could go around and see all the companies, and whether they were mature enough to use these models or not. (4:19)

Abouzar: It was fun, I think. Fun years to start actually. So yeah, I started that program. It was a two-year program. It was not a PhD, but it was research plus practical work. (4:48)

Alexey: We call it doctorate. It is confusing, right? (5:01)

Abouzar: Yeah, it is confusing actually, but it is research plus practical work. They basically pay you to study. That was an amazing opportunity. (5:08)

Alexey: Ah, that is a good deal. I did not have this deal. What they offered me was a partial scholarship. Yeah, that was not enough. (5:14)

Abouzar: Yeah, this was also like you get the minimum wage. (5:26)

Alexey: Yeah, that is really cool. (5:32)

Abouzar: That is awesome. It was enough, I think. For a student life it was enough. During that program, basically, you worked with a lot of Dutch companies that kind of sponsored the program. (5:38)

Abouzar: Like Philips, ASML, and we worked with Hendricks Genetics, they were breeding turkeys and stuff. It was a lot of cool companies that you got to work with, even municipalities. A lot of companies actually worked very closely with TU Eindhoven. So it was really cool. (5:45)

Moving to Europe for data science

Abouzar: Then I went to Efteling, then to Bol.com, and now Tesla. (6:17)

Alexey: Okay. At the end, you said you went to… oh wait, after that you went to Efteling and then Bol.com. I have no knowledge about this link, but I see it is an amusement park. What is this? What did you do there? Because I also, sorry for interrupting you, I also worked at an amusement park. (6:23)

Alexey: I was a ride operator. I let kids in and made sure they stayed safe. That was my job. I think yours was different, right? (6:50)

Abouzar: A bit different. So the story basically starts from my second year. In the second year of our program, we had to have a long project. For that one-year project I either had to go with the banks, in collaboration with the police, to work on money laundering. Because I did graph theory and a couple of projects like that, they thought maybe this was suitable. (7:02)

Abouzar: The second option was to go to Efteling, the amusement park. I did not like the banking option, so I went with the second choice. At first, I was quite lost. I thought, what the hell am I doing here? How did I end up here? My background was technology and tech companies, and now I had landed in an amusement park. (7:36)

Abouzar: So I started to understand what the problem there was that I could help with. You would be quite shocked when you land in an amusement park and think, what kind of data could they possibly have that I can design a model or train a model on? It started from there. Their main problem was crowd control. (8:01)

Alexey: I imagine this is like Disneyland, where you have many rides, and for each ride you stand in the queue, wait your turn, then ride for 5 seconds, and then it is done. At least that was my experience in Disneyland. Did you have a similar problem there? (8:47)

Abouzar: Yeah, exactly. I think you know the business better than me because you were on the floor. Efteling is the biggest theme park in the Netherlands, and in terms of revenue maybe the second or third in Europe, after Europa Park and Disney. (9:10)

Abouzar: It is very popular. In the Netherlands or in the part of Germany close to the Netherlands, everybody knows it. The significant thing about Efteling is that every ride has a story. Almost every ride has a story behind it. (9:36)

Abouzar: It is not just a ride where you get scared or get an adrenaline rush. Everything has a story behind it. For example, a roller coaster is about a miner who got stuck there, or it was haunted. That was the cool part, and the thing that triggered me to think, who came up with this idea? (9:58)

Abouzar: The most interesting and impactful part for me was learning about a guy called Peter Reinders. He was the mind behind creating the motors and movements, and putting mechanical stuff together to bring the magic to life without showing the technology. It was presented in a way that looked magical. (10:33)

Abouzar: I found myself interested, not saying I am an innovator of the time, but he was a big Dutch innovator. He laid the foundation for all the innovations that you can see in the park. For example, in Efteling, Snow White is just laying there behind a window. Every kid sees her, and she is breathing. (11:08)

Abouzar: That small detail of breathing adds magic when children watch her. These subtle details are really important, and it can be the same in our jobs as well. When presenting something, users do not need to know how complicated the backend is. It just needs to function well. (11:48)

Working with theme parks and crowd modeling

Alexey: That is the story of all software engineers, right? You do not need to know how it works. But what did you do there? Did you do analytics and data science and machine learning? (12:18)

Abouzar: Yes. The problem was how they could control the crowd in different places in the park, or how they could drive groups of people to other places. Usually, people rushed to the roller coasters, but there were also story rides, family-friendly areas, and attractions for kids. (12:32)

Abouzar: The goal was to create a recommendation system to plan their day and recommend their next move in order to avoid long waiting times. That was the ultimate goal, to improve the experience. Because if you go to an amusement park and all you do is spend time in queues, it is not fun. (12:59)

Abouzar: In order to do that, we needed to predict how long the queue would be. You also needed to know the ride capacity. And you could also guess how crowded an area of the park would be based on data such as transactions at restaurants or stands selling drinks. (13:36)

Abouzar: We had a couple of models. One came from transactions, others came from the rides. These gave us an index of how crowded each place was. We used that in another model that tried to predict and recommend the next best move for each group. (14:09)

Alexey: From the user point of view, did they have an app, and it said “go here now, go there next”? And then this way you redistribute the crowd so they do not all stand in one queue? (14:32)

Abouzar: Yes, exactly. (14:50)

Abouzar: Efteling had a very popular app. I think back then at least 60 percent of the people used the app. (14:55)

Alexey: Makes sense. Especially if you advertise it like “do you want to wait less? Download our app,” then of course everyone says yes. (15:00)

Abouzar: We gave another incentive. When we wanted everybody to do the survey, we told them “do you want a free coffee?” Then they installed the app, did the survey, and got their free coffee at the start. (15:06)

Alexey: What you are describing reminds me of one of the classes I had in my bachelor studies, a modeling class. I do not remember the tools, I think it was SPSS Modeler. We modeled supermarkets, where people came at random times to cashiers, and we wanted to see what was the most effective way of dealing with customers. (15:30)

Alexey: We tried different arrangements for cashiers. What worked best was having one queue for all the cashiers. Then each person went to whichever cashier opened next. That worked best in our modeling. Did you use a similar approach? Or was it something completely different? (15:55)

Abouzar: Actually what we did was a bit different. From the survey, about 3,000 people participated and we got all the moves they made through the day. That was cool. We had a bit less than 3,000 variations of routes people would take, based on their preferences. (16:40)

Abouzar: For example, a group of two could specify in the app if they had lunch or not, whether they were daredevils or just wanted a chill day. So we had all these variations for each group of people, and we built a model based on that. If you gave your preference, the model would place you close to one of these variations. (17:20)

Abouzar: It gave a probability for a list of attractions you could take. On the app we showed the first suggestion, the one with the highest probability. It was simple but effective. (17:50)

Alexey: Yes, like many things, the simplest solution often works really well. (18:11)

Abouzar: Yes, especially since you do not know who is going to the park. It could be a 15-year-old or a 60-year-old not familiar with technology. (18:16)

Lessons from ride and visitor data

Alexey: Then you worked at Bol. I do not live in the Netherlands, but I know about Bol. First, I know because people from there often attend Berlin Buzzwords. I do not know if you ever went, but Berlin Buzzwords is a conference in Berlin about search and other hot topics like machine learning. This year it was AI, but typically search. (18:29)

Alexey: People from Bol came every year to Berlin Buzzwords, giving talks about the challenges they solved. That is how I knew about Bol. It is a big company, kind of like Dutch Amazon, and they started with books and now sell other things as well. (18:59)

Abouzar: Yes, they started exactly like Amazon, selling books, but now it is everything. (19:15)

Alexey: And you worked there. What did you work on? (19:22)

Abouzar: I worked specifically on the favorite brand carousel you could see in the app and on the webshop website. That was my biggest product there. Users could browse their favorite brands. (19:28)

Abouzar: Back then there was a big push from managers to integrate brands into websites and apps. The algorithm basically showed what was the favorite brand at the moment and whether it was related to you. (19:55)

Alexey: And brands could be book-related? Like if I read books then my favorite brands could be publishers? (20:21)

Abouzar: Kind of, yes. But it could be anything. (20:32)

Alexey: Like Nike, right? (20:37)

Abouzar: Yes, it could be a clothing brand, book brand, publisher, or anything. That was my biggest project, though I also did smaller things. (20:39)

Alexey: I just open Bol and I see, well, first of all, everything is in Dutch. But since I speak German, it is kind of understandable for me. I see watches, coffee machines, washing machines, smartphones, lamps, furniture, basically everything. (20:51)

Abouzar: So yeah. Yeah. Yeah, it has everything. (21:10)

Alexey: The film, stools. Okay. So that is why, like let’s say if we talk about clothes, it could be Nikes or Adidases, right? (21:16)

Abouzar: Yeah. Exactly. So it could be more related to what you purchased before, but also it has an element of how popular the brand is. Because maybe you bought a screen protector for your phone, but it is not your favorite brand, although it is popular. It is not really your favorite. (21:30)

Abouzar: Favorite is more in the realm of sociology. It is where you would really spend money without caring about how much you spend. Apple is an example of that because it has a strong fan base. People buy without thinking about the cost, since it is a perception of value. (21:56)

Alexey: But let’s say I like Nikes because they are really comfortable. If Nike increases the price, I might think, yeah, maybe I check these other ones. Does that mean it is still my favorite brand or not? (22:24)

Abouzar: I think it then again depends. (22:34)

Alexey: Like when it comes to shoes, I would prefer Nike because I know when I wear the sneakers, they just fit my feet. (22:39)

Abouzar: Yeah. What counts as favorite is probably also behavioral. You will maybe check trends from Nike, check their latest releases, or you favorited them. (22:46)

Building recommendation systems at Efteling

Alexey: I do not do any of these things. If I need shoes, I just go check what models are available. Usually, I go with the same model. But what I really hate is that they retire models. The model I like, I want to buy, but it is no longer available. I am like okay, what do I do now? (23:06)

Abouzar: Yeah, it happens to me as well. I think Zalando does it best in that category. It has a really nice way of showing similar products, probably with a good model in the background. They present alternatives very well. (23:29)

Abouzar: I think at Bol we also had a similar products carousel. But Zalando is doing really well in terms of exploring similar products. (23:50)

Alexey: And since you worked in recommender systems, was it something simple or more complex, like neural networks? (24:03)

Abouzar: I think the biggest challenge in recommendation systems is how do you validate them, really? (24:16)

Alexey: Yeah. Right. A/B tests, or what was it? (24:23)

Abouzar: Yeah. Then A/B test is also biased depending on how you define the A/B test. The real performance is challenging, because how do you say, how much did I sell from the recommendations, or how many clicks did I get from the recommendations I showed? (24:31)

Abouzar: But then again, was it really what the users wanted? Or could we have done better? So validation is tricky. One interesting thing about the favorite brand project was that we thought about this problem and asked, how can we test it? So we created something like a swiping game. (24:49)

Abouzar: We showed this swiping thing to all the employees involved. That is how we tested the algorithm, and we were convinced it worked. Then we released it. Maybe I can share this. Yeah, okay, I hope you can see my screen. (25:15)

Alexey: Yeah. But we will release this as audio only. So if you could also describe what is happening on the screen, that would be helpful. (25:42)

Abouzar: You can see my screen. (25:48)

Alexey: Yeah. Yeah, it’s loading. (25:53)

Abouzar: Okay. We tried to make it fun by adding Star Wars loading gifs and things like that, so it would not be boring. Before the employees saw this page, we did streaming processes because we did not want to process three million or six million users. (26:01)

Abouzar: We just wanted to do it for the employees, otherwise it would be costly. And we did not know which users were employees, so we had to do calculations on the fly just before they saw the page. And it took like 45. (26:15)

Alexey: So the experience for the users was that they go to the website, and if they are employees, then they see this? (26:29)

Abouzar: It was an internal website. Yeah. So we did all the processing there, and they could easily say, okay, this is my favorite brand, or what you are showing me is not my favorite. (26:41)

Alexey: So right now, can you describe what you show? Like, do you want to see Stabilo, Ubisoft? (26:53)

Abouzar: Exactly. For example, on a card we had three buttons that said, it is not my favorite, I like it, or this is my favorite brand. (27:00)

Alexey: It seems very random. Stabilo creates markers, Ubisoft is a game software company, Xiaomi produces what? They produce phones and wearable devices, right? (27:11)

Joining Bol.com and the Dutch e-commerce industry

Abouzar: Yeah. But your way of buying stuff is not just books from Bol. You usually buy a wider variety. (27:26)

Alexey: But like if I need markers and I see Stabilo, I will go with Stabilo. I do not know how to pronounce it exactly, but I know they are good. If there are brands I do not know, I will just go with Stabilo. (27:43)

Abouzar: Yeah. Basically, you want to see the perception about the brand. Or even if people know the brand. For example, 999 Games, this is a board game company. If you are into board games, you obviously know them. (28:00)

Abouzar: If you like Lego and things like that, you would swipe right for them. Employees swiped left or right for individual brands, and we could see if our favorite brand algorithm worked for them or not. (28:19)

Alexey: There is a comment from Adonis. Hi Adonis, by the way. He says it looks like Tinder for brands. (28:37)

Abouzar: Exactly. (28:40)

Alexey: I guess we all know where you got the inspiration from. So that was the approach, but how did it… So I worked at a similar company. It was not e-commerce, it was an online marketplace, but the idea is similar. (28:43)

Alexey: You have items, you have people to recommend them to, and you develop a recommender system. You already have a basic recommender, then you improve it. You split traffic into test and control groups, one sees the new algorithm, the other the old one. (28:59)

Alexey: Then you check metrics. In our case it was conversions. For that company, conversion meant when people wrote to each other. For example, on a marketplace site like Marktplaats in the Netherlands, you want to buy a loudspeaker, you write to the seller, that counts as conversion. (29:29)

Alexey: We measured if in the test group there were more conversions than in the control group. If so, the new algorithm was good and we rolled it out. So that is the basic A/B test. Why did you need to do this Tinder for brands instead of the usual way? (30:07)

Abouzar: Because there was nothing similar before. (30:31)

Alexey: Okay. So your control group would be nothing, and the test group would be this new system? (30:37)

Abouzar: Yeah, basically. Or we could have also aggregated purchases or views by brand and then shown random items to test against this one. That could have been done too. But this was really about convincing our team that this could work. (30:43)

Alexey: Makes sense. It gave you internal agreement that the idea was worth investing in, so you could spend a quarter or so developing it. (31:08)

Alexey: How did you use the results? Swiping is fun, but at the end, what was the outcome? (31:19)

Abouzar: Basically, we measured how accurate we were at showing the brands. A couple of items in the list were not actually their favorite brand. At the end, we could say that about 85 percent of the items swiped were actually their favorites. (31:39)

Alexey: I see. So the algorithm already knew their likely favorites, like Lego, Nike, Xiaomi. Then you added three more brands that were not in the algorithm’s output. Ideally the users would swipe the recommended ones as favorites and the extra ones as not favorites. (31:57)

Abouzar: Yeah. That is how we validated it again, showing the model worked, before releasing it as something brand new. (32:35)

Product and brand recommendation logic

Alexey: And for the user, they see a brand, click, and can buy stuff from it, right? (32:49)

Abouzar: Yeah. Exactly. The purpose of this component was that they clicked the brand and saw featured or popular products from that brand. Engagement with brands on Bol would go higher. That was the point. (33:02)

Alexey: What do you do now? (33:15)

Abouzar: Currently, I work as a data engineer at Tesla. (33:21)

Alexey: Sorry, at Bol you worked as an ML engineer or as a data engineer? (33:27)

Abouzar: So I did both actually. I did data science plus data engineering. When I went there it was like okay, we have this project, and I was interested to work on it. That was it. (33:35)

Abouzar: I think that mix comes from my background. You never fully transition. You carry your background with you. I have a software engineering background, that is the story of my life. Whether you want it or not, you bring interests from your previous work and involve them. (33:55)

Abouzar: That was the same for me. At Efteling with the survey app, I wrote the Android app even though I was the data scientist. It helped data collection. I pick up those tasks because I am also interested in machine learning, and I like deploying and training models. (34:21)

Abouzar: But you need to really understand if it will help, or if you are just making it complicated for nothing. If it helps, I will do it. (34:48)

Alexey: So the idea is, even though you are a data engineer, you can do pretty much everything needed. (35:16)

Abouzar: Yeah. At Tesla that is the case. It is very hands-on, and you just create stuff that is functional without worrying about making it fancy. Maybe in other software companies it is different, but I also want to state that I am not speaking on behalf of Tesla. (35:24)

Alexey: Yes, that is important. But can you actually talk about the things you do at work? (35:47)

Abouzar: Not so much to be honest. I can generally talk about who I am and my background, but I cannot talk about projects at Tesla in detail. (35:54)

Experimenting with "Tinder for brands"

Alexey: What is interesting is your mix of skills. For me it was similar. My first job as a data scientist, I was hired because I knew Java. I was a software engineer working with Java, and even though I had done my master’s, they only cared about my Java. (36:09)

Alexey: The data scientists there used R and Python but did not want to touch Java. So they needed someone who knew it, and I did. (36:43)

Abouzar: Yeah, that is exactly how it is, I think. Sometimes when fields advance, they diverge. At first there is just software engineering, then it becomes backend, frontend, DevOps. Later it can come back together again. (36:49)

Abouzar: For data scientists, if you do not know deployment, or you only know statistical methods but cannot transform data, is that enough? I think you need to know all of these together. (37:22)

Alexey: So when I asked what you do now, you said data engineer. Do you identify as data engineering, or more of a generalist? (37:42)

Abouzar: I think what I am best at is being a machine learning engineer. My day-to-day tasks are maybe 60 to 70 percent data engineering. The last 30 percent is software engineering, machine learning engineering, and some data science. (38:02)

Abouzar: So yeah, it depends on what my employer wants, to be honest. (38:23)

Alexey: I understand. For me, when I was a data scientist, the issue was that many data scientists were not good at deploying models, and they did not want to do it. Because of my background, I could fill that gap. For example, I did not know Kubernetes before my job, but I learned it because the team needed someone. (38:31)

Alexey: Sometimes data scientists cannot always pick these things up. But you said you just do what the employer needs. How does that work in practice? There are some things you want to do but cannot, and some things you do not want to do but still need to. How do you balance that? How do you prove yourself so you get more responsibilities? (39:19)

Abouzar: Yeah, that is a good question. I do not know the exact answer. Sometimes you need to get yourself out of the trap. You might think, I do not need to do this dashboard, but then see that it is actually an opportunity. (40:07)

Engagement metrics and product validation

Abouzar: Opportunities are things other people might not see, but you do. Then you feel obliged to take it. You can do the standard solution, like solving with a script, or you can say, no, a model will improve it a lot. That is how it works for me. (40:26)

Abouzar: If I see there is an opportunity to improve something exponentially, I try to do it. Of course, there are boundaries. You need to look at your time and resources. But most of the time, it is about underpromising and overdelivering. (40:58)

Alexey: So, okay, that is a good approach, right? Because if you overpromise and underdeliver, it is way worse. (41:19)

Abouzar: That is way worse. But it takes a lot of effort. It takes a lot of energy from you. Yes. (41:25)

Alexey: And you are still doing some machine learning, right? (41:36)

Abouzar: Yes. I always engage myself with that, and now with LLM agents and all those things being used across companies, including Tesla. It is a nice opportunity to improve your tasks as well as the platforms you are providing for colleagues. It is great. I think it is a great time to be in the field. (41:43)

Alexey: You have been in the industry for quite some time, so you also saw this rise of LLMs and AI in recent years. How did your work change with that? Now the challenges are different. The models we deploy, especially LLMs, come with their own issues. What are these challenges for you, and how do you deal with them? (42:08)

Abouzar: I think first it drastically changes things. Sometimes I am shocked, like wow, this task would have taken me four days, but I can now solve it in a couple of hours. Especially for developers, that is amazing. The pitfall is when everybody starts thinking AI can solve everything. (42:34)

From ML engineering to data engineering roles

Alexey: Are you talking about using AI for creating code, or using models to solve the problems themselves? (43:02)

Abouzar: I think mostly creating code. (43:15)

Alexey: Okay, so coding. The correct way to say it is AI-assisted development, right? (43:20)

Abouzar: Yes. But I like web coding more. (43:25)

Alexey: Web coding means you just let whatever code is produced get shipped. And then sometimes it drops your database, and you are like, oops. (43:32)

Abouzar: Not that. AI-assisted development is different. (43:44)

Alexey: At least this is how I understand it. Web coding is something that kind of works, but you do not care what is inside. (43:49)

Abouzar: Yes, true. But it can help you develop code, refactor, and handle documentation. To be honest, the most significant usage that LLMs have is really in these areas. (43:55)

Alexey: For me, I use them for coding but also for editing and writing texts. This is how I started using them more than two years ago when GPT appeared. Back then it was not that useful for coding since GPT-3.5 would produce code that did not always work. (44:18)

Alexey: Now with today’s models you can go to platforms like Replit or Lovable. You just say, “I want to build an application where I can swipe brands left and right and see which one I like,” and a few minutes later the app is deployed. (44:47)

Abouzar: Yes, I saw that. I have my opinions about it. I think it is hard to believe sometimes. To me, the biggest advancement would be physical AI. But I agree with Yann LeCun from Meta who said maybe AGI will not just come from LLMs. (45:34)

Alexey: Yes, because you also need hardware, physical components. For example, cars need to make driving decisions. This is where AI could be useful. Hypothetically, since self-hosting LLMs is much more complex than before, did you find any helpful tools? (46:06)

Abouzar: For inference, yes. On weekends I work on these projects. Some vision-language models are optimized enough that you can run inference on a Raspberry Pi, though it is slow. I also saw Nvidia Orin as a development kit optimized for inference. Even Mac Minis are amazing for this. (47:05)

Abouzar: There are projects where you can cluster them together and get really good performance in tokens per second. Hardware is catching up, and I think it will only get easier in the future. (47:55)

Alexey: I checked Nvidia Jetson Orin and it costs around 2000. A little more expensive than Raspberry Pi. (48:21)

Abouzar: Really? I thought it was around 700. (48:28)

Alexey: There is another version, Jetson Nano Orin, which has 8 GB instead of 64 GB, costing around 350. Still more expensive than a Raspberry Pi. But when you said you were deploying LLMs on Raspberry Pi, I was impressed. (48:33)

Abouzar: That is useful if you have a small team of maybe five developers. You do not want to pay 200 euros per month per user. Instead, you deploy one model on an Orin device that they all query occasionally. It works. (48:53)

Alexey: Yes, because 8 GB is enough for some smaller LLMs, like those recent Chinese models. (49:25)

Abouzar: Yes, especially specialized models for coding. You can put that in and handle coding questions. It can bring costs down for startups. (49:38)

Alexey: Do you know any models like that? (49:53)

Abouzar: I think the 2.7 billion parameter LLaMa model is working well. There are others too like Gro, and cloud ones are amazing, but for open source, LLaMa is good. (49:59)

Alexey: Do you need to fine-tune them for coding, or are some already pre-trained for that? (50:11)

Abouzar: I think Gemini and others have code-specific versions. Replicate is a good place to check these. (50:35)

Alexey: How do you spell Replicate? (50:56)

Abouzar: Replicate, yes. (50:58)

Alexey: What is that? (50:59)

Abouzar: It is my favorite website right now. (51:00)

Alexey: You mean replicate.com? Run AI with an API? (51:03)

Abouzar: Yes. They host models and provide APIs. They charge per request. They have a wide range of models for video, photos, and more. (51:09)

Alexey: I see, that is interesting. I am adding it to my bookmarks. (51:32)

Abouzar: It will probably give you a lot of application ideas. (51:37)

Alexey: We have a question from Adonis. He asks, what factors had the greatest impact on advancing your career? (51:45)

Hands-on skills at Tesla and industry expectations

Abouzar: Hi Donis, I know him, he was my classmate. I think finding a north star was key. For me, I was very motivated to work for Tesla, a company with vision. A vision that drives both the technology and the business is inspiring. (52:04)

Abouzar: A big vision motivates everyone and drives them forward, even when the goals seem impossible. That had a great impact on me. (52:34)

Abouzar: I also think about impact at this point in life. Earlier, I would just code all day, but now I think about who I can impact and where my contribution makes sense. That shaped my career choices. (53:25)

Alexey: Was it difficult to get a position at Tesla? (54:03)

Abouzar: Yes, it was quite difficult. (54:10)

Alexey: How did you prepare? What did you need to know for the job? (54:15)

Abouzar: You do not know exactly what they will ask. I looked for other people’s interview experiences and questions. I printed a list of what data engineers should know and studied everything, like distributed systems. That is how I prepared. (54:22)

Alexey: I asked GPT to generate Tesla data engineering questions. Back then ChatGPT had limitations though. How long have you been working at Tesla? (55:29)

Abouzar: Almost two years. (55:47)

Alexey: Back then it could not search the internet. But it suggested SQL, scripting, ETL, architecture, behavioral questions. Was coding emphasized, like LeetCode? (55:52)

Abouzar: Not much. That is more of a Google thing, not here. (56:18)

Alexey: So it was more system design and architecture questions? (56:24)

Abouzar: Yes, mostly architecture, scripting, and those things. (56:29)

Alexey: I imagine they asked about cars producing sensor data and how you would capture and store it for analysis. There is already so much to talk about in such a question. (56:38)

Career growth, learning, and advice

Alexey: So another question. You said if you see an opportunity to improve exponentially, you take it. But you also have everyday tasks. How do you prioritize? (57:43)

Abouzar: What do you mean by opportunity? (57:50)

Alexey: We were talking about tasks as a data engineer. You also sometimes expand into things like Kubernetes, like I did. I see these as opportunities, but they compete with routine tasks. How do you prioritize those improvements? (57:58)

Abouzar: It goes back to underpromising and overdelivering. You put in more effort, maybe even in your free time, to deliver results that can really improve things. Sometimes it takes personal time, but the outcome is worth it. (58:59)

Alexey: I like that answer. Underpromise and overdeliver. You create a buffer, and when it works out, it looks great. (59:37)

Abouzar: Yes, exactly. It is not about the minimum. It is about something acceptable, plus delivering more than expected. That is how it usually works for me. (59:51)

Alexey: Abouzar, thanks a lot for joining us today. And thank you Adonis for your questions. It was fun, I really enjoyed the interview. Thanks everyone for joining us for today’s session. (1:00:10)

Abouzar: Thank you for having me. It was fun actually. (1:00:35)