Understanding the AI Engineer Role | Nasser Qadri

The transcripts are edited for clarity, sometimes with AI. If you notice any incorrect information, let us know.

Transitioning from Social Science to Software Engineering

Alexey: Hi everyone. Welcome to our event. This event is brought to you by Data Talks Club, which is a community of people who love data. We have weekly events. (0.0)

Alexey: Today is one of such events. I am not doing the usual presentation because we are starting a bit late. I wanted to jump immediately into our interview today. (0.0)

Alexey: This week we will talk about the AI engineer role. I am quite interested in this role. Lately, you may have noticed that we are putting out more content about this and I want to talk about this role today with our guest Nasser. (15.0)

Alexey: We want to discuss what the role looks like in practice, what kind of work it involves, and how people transition into it. Nasser is an engineering manager, formerly an AI engineer at Google. Previously, he held data science leadership roles at Quest Diagnostics, Deloitte, and Gallup, working across domains such as healthcare, the public sector, and national security. (27.0)

Alexey: Before moving into industry, Nasser completed a PhD in politics and international relations at the University of Glasgow. His background combines social science research, data science, and applied machine learning. Welcome to our interview. (55.0)

Nasser: Thanks for that introduction. I have actually got my University of Glasgow shirt on today. I do not know if your listeners or watchers are affiliated, but are you based there out of Glasgow? (1:13)

Alexey: I am now actually based out of Zurich. (1:27)

Nasser: I started my career at Google in their New York office, but then recently transitioned to a new role that brought me to Zurich. (1:32)

Alexey: Okay. How is your German? (1:38)

Nasser: It is non existent right now. (1:46)

Alexey: Do you need German in Zurich? I have never been in Zurich. I have no idea how it is there. (1:51)

Nasser: It is helpful as all things, particularly to acclimate. Every once in a while you will end up in some remote region in Switzerland where English is not as common. For the most part, if I started to try to speak German, most people would probably tell me to just speak in English. (1:51)

Alexey: I have the same problem in Berlin. I try to practice German, but people would just speak English back to me. I guess I will never learn it. Your background is social science and politics? (2:09)

Alexey: How come you ended up being an AI engineer? How did your career look? (2:21)

Nasser: Great question. Interestingly enough, my entire academic background has been in the social sciences. I did my bachelor’s in economics, my master’s in international relations, and my PhD in politics with a big focus on international security. (2:37)

Nasser: Sometime after I had done my bachelor’s, I had always been interested in computer science. It was a track that I had explored in undergrad and decided that I would try something that was a little bit different from my expertise. I had a programming background and I had some fundamentals. (2:55)

Nasser: In university, I wanted to try something completely different. Of course, right after I graduated, I took on a software engineering role because that was where the money was. I stayed in that field for a long time. (3:21)

Nasser: When I was going back to graduate school, I just had personal passions and personal interests in foreign policy, diplomacy, and the general field of international relations. I figured I was already doing work as an engineer, so why not try to focus my academics on things that I was more interested in or had a bigger personal passion for? It was convenient because at the time that I was going to grad school, the field of data science was just emerging. (3:39)

Nasser: Natural language processing was becoming common. While I had not quite broken through as much in the social sciences, there was more of an interest in doing text as data and doing analysis using tools that maybe did not have the same computer science engineering rigor. There was some overlap, so I tried to tailor a lot of my research towards using those tools from the data science and statistics toolkit. (4:09)

Nasser: After grad school, I took on some roles in data science. It was a very good confluence between the engineering background that I already had and the statistical skills that I developed in grad school. It pivoted from engineering to data science and now back into the software engineering space. (4:44)

Alexey: It is an interesting hobby that turned out into a PhD with political science. Do you use any of this stuff right now? I assume that all the social sciences are heavy on statistics because when you need to do some sort of experiments or surveys, you have to have the statistical rigor to make sure that the experiments you conduct are sound. (5:06)

Alexey: Do you use these skills right now as an AI engineer or as a manager? (5:31)

Nasser: Absolutely. I can talk a little bit more about where I see AI going and the overlap with data science. I think AI, the way that we speak about it right now, has been conflated with the use of generative AI. (5:49)

Alexey: For me, when I say AI, mostly I mean generative AI. These Large Language Models are what I mean because of course if we talk about AI in broader terms like machine learning, all these things fall under the AI umbrella. This is what we studied at university. (6:06)

Alexey: There is this big circle for AI, a smaller circle for machine learning, and an even smaller circle for deep learning. Most of the time now when people talk about AI, we talk about things like Large Language Models or ChatGPT. We talk about these generative models. This is what I mean by AI, and probably also an AI engineer most of the time means a generative AI engineer working with these tools. (6:13)

Nasser: Those terms have been interesting because I feel like I have seen the same thing happen in my career with other terms. Data science was very science based and scientific. It was a lot of stats. (6:42)

Nasser: In my career, I also discovered that people who were building dashboards and Tableau things, what I would have generally considered to be business intelligence or data analysis, were co-opting the term data science for that as well. I think AI is similar now. AI ten years ago meant something very different to me than what it does today. (7:02)

Nasser: If we use that narrow definition of AI as being more generative AI and the concentric circles, that was very much born out of data science as well. That was very much born out of transformer technology. (7:22)

Applying Statistical Rigor to Generative AI Evaluation

Nasser: I see some rhyming effects where the way that we assessed our data science models or our traditional machine learning models is starting to happen now with AI models as well. (7:45)

Nasser: We are trying to bring in evaluation metrics, whether it is precision, recall, or accuracy. If we are using agents for classification tasks or for things where we have substituted traditional machine learning, a lot of the evaluation still rhymes with the way that we would evaluate data science models. I think that aspect of the statistical background has continued to be helpful in AI engineering. (7:55)

Nasser: The other element that is helpful is that so much of social sciences is about building the fundamental domain knowledge of an area to be able to do deep research on it. In my case, there were a couple of case studies that I was interested in with respect to political communication or foreign policy decision making. That required me to just become an expert in those domains in order to be able to run analyses or run experiments against that. (8:26)

Nasser: So much of data science now and AI engineering requires the same. You listed off different industries that I have done work in: healthcare, financial services, and national security. In all of those cases, I have had to have enough knowledge in those domains to be able to speak with the experts and to have a productive conversation. (9:16)

Nasser: Social sciences research equips you to do your literature review and your diligence in becoming a subject matter participant. I would be cautious about saying expert, but it helps having at least the requisite knowledge to be able to talk about these things with the experts so that you can build your models effectively. (9:41)

Alexey: When I look at the people who currently work as AI engineers, it is interesting to analyze their background and where they were coming from. There is already a question about what background you need to be an AI engineer. From what I see, there is a big category of people who worked as machine learning engineers. (10:12)

Alexey: For them, the transition to AI engineer is the smoothest one. They just replace PyTorch with calls to OpenAI or Gemini. They are already engineers and they already work in machine learning, so this is very related. They just need to get the basics of generative AI like the tooling. (10:39)

Alexey: Data engineering is a little bit more complex but still probably the second smoothest transition. We also have data scientists and people who work in research. They also transition to AI engineering, but for them, the engineering component is missing. (11:04)

Alexey: I guess for you that was also the case? You come from social science and you studied computer science and worked as a software engineer, but then after that, you spent a lot of time working on social sciences for your PhD. You are probably a mixture of both. (11:16)

Alexey: This is the second background I see: people who work in sciences, people who work in research, and also data scientists or analysts. They are also interested in becoming AI engineers and they need to upskill themselves in engineering. How exactly did this look for you? (11:40)

Alexey: Would you consider yourself more like an engineer or more like a researcher? For you, the transition was very smooth and you just needed to learn the tooling? (11:53)

Balancing Research Mindsets with Engineering Speed

Nasser: This is a great question. I would probably argue that I have researcher envy, which means that I am probably more of an engineer than I am a researcher. I very much considered the researcher academic route and decided at some point that I like the fast pace of the engineering world. (12:13)

Nasser: I made that switch. For that path from academic social science or academic stats to working with AI or becoming an AI engineer, there was a period where it was easy to go from scientific and less engineer heavy experimentation and modeling. We went from using traditional machine learning tools to coming into the world of agents. (12:34)

Nasser: For a long time, working with agents and even now, there is still some element of working with AI tools and Large Language Models where you are simply just making an API inference request. You pick your model of choice, generate content, and you send in a prompt. The current phase and what is becoming more common is introducing a little bit more software engineering rigor to agent development. (13:16)

Nasser: Thinking about tools like ADK, MCP capabilities, there is more software engineering beyond simply just making an API request to a large language model. It will be beneficial to have that engineering rigor and some of that development rigor that is generally in excess of simply building machine learning models and experimentation in your Jupyter notebook environment. We will see that more often. (13:50)

Nasser: As agents become more standardized, the capabilities mature, and so does the tooling around it. I saw this happening with machine learning models and the development of MLOps. I think we will see the same thing with agents and agent ops. (14:32)

Alexey: For people coming from the research background, their superpower is this experimentation mindset. They have an evaluation mindset where they conduct an experiment and know how to evaluate it. They collect a data set and run the system through it to establish a metric. (14:57)

Alexey: Once they establish the metric, they can improve it. People who come from the engineering background do not necessarily have that. For you, as someone coming from this sciences background, was it natural to think about evaluating the tools you need to use? (15:16)

Nasser: That is an excellent description of this. It is that sort of experimental mindset. The way that I used to describe the world of data science versus the world of engineering is that engineering is very black and white. (15:42)

Nasser: You have a goal, and while there are multiple ways to get to that goal, when you get to it, you know you are there. If your target is to come up with the number four, you can add two and two or take the square root of sixteen. You will get the answer four and you will know that you have done what you were supposed to do. (15:57)

Nasser: The world of data science and AI is just murkier. It is a lot grayer and there is room for error. It is certainly not always going to be deterministic, which is partially the benefit. (16:21)

Nasser: There is a lot more gray area. Not only do you have to accept an experimental mindset where you are setting hypotheses and trying to evaluate based on metrics, you also have to accept that it is not black and white. There is a lot of gray in between and you have to tweak the parameters to get the outcome that is good enough. (16:33)

Alexey: We already spoke about AI engineers and hinted at what they need and what kind of backgrounds they have. You also said that you need more software engineering beyond just making API calls. Do you have a definition of what an AI engineer is, who that person is, and what their role is in a few sentences? (17:14)

Nasser: That is hard because it is evolving so fast. I once had a notion of what a data scientist was. Rather than defining exactly what that role is, I think thinking about what the day to day looks like is important. (17:44)

Nasser: One is the experimental mindset that we have already talked about. You have to accept that you are not building a deterministic system. There is going to be some room for error, or probably a better spin on error here is creativity. (18:02)

Nasser: Being an AI engineer requires some thinking beyond just the software and engineering components. There is some level of domain knowledge that is always beneficial to have. Healthcare is a really good example. (18:34)

Nasser: I have done a little bit of AI engineer work in the healthcare space. As an AI engineer, sometimes you have to thread the needle between how you instruct this system even when engineering prompts. You have to instruct the system in using the right terminology and setting up the right evaluation framework, which inherently requires being able to speak the language and understand the terminology in your domain. (18:58)

Nasser: That is something very different from traditional software engineering. (19:33)

Alexey: Different from what? Do you compare it with traditional software engineers or data scientists? (19:51)

Nasser: I am thinking more along the lines of a traditional software engineer. For data scientists, they already need to have this domain knowledge and they already need to have this experimentation mindset. From what you described, it is not much different except in one case you talk to business and train your XGBoost model, and in the second case, you talk to business and make API calls to a provider. (19:57)

Comparing AI Roles in Big Tech vs Startups

Nasser: Hearing you say that out loud, maybe the best description is that the AI engineer is somewhere in between the traditional software engineer and the data scientist. It requires the engineering rigor or benefits from the engineering rigor that the software engineers already have. It also requires the experimental mindset and the evaluation mindset that data scientists have. (20:27)

Nasser: Data scientists did not always have that traditional software engineering and computer background as well. (21:00)

Alexey: When I asked you about the background, I listed two previous backgrounds of people who now work as an AI engineer. One is the person who is already an engineer, so they need to have this experimental mindset and domain knowledge. The second is more research oriented, and they need to have more software engineering rigor. (21:06)

Alexey: Am I missing anything else here? There is a question from an electrical engineer asking if they can become an AI engineer. They do not necessarily have the software engineering rigor. (21:39)

Alexey: I apologize if I am making some assumptions that are not right, but probably they will still need to be doing some catching up to be at the level of software engineers. They do not necessarily have the level of data scientists. Is the road to AI engineering for people coming from traditional engineering longer? (21:57)

Alexey: Is it actually possible? Have you seen people transition into this role from those kinds of backgrounds? (22:15)

Nasser: I have seen a lot of my academic peers transitioning into this role. I would argue that the transition from an electrical engineering or some STEM background to AI engineering is a smaller distance than the transition from social sciences. Given that I have seen the latter frequently enough, I think this is something that is within the grasp of most people. (22:27)

Nasser: As someone that has spent so much time there, there was a period when I did not have the software background or the computer science background. Yet, I still am in this space. Everything is achievable and everything is learnable. (23:11)

Nasser: We are kind of lucky to have so many resources. Maybe the bigger problem is that we have too many resources to learn this stuff and it is hard to know where to focus your attention. I would very much encourage the electrical engineer that is interested in this to pursue that path. (23:22)

Nasser: I think it is a smaller distance than they may think. (23:49)

Alexey: Thanks for the response. You mentioned that we do not have the problem of lack of resources, but the opposite problem. You open Twitter and you see tons of new courses, tons of new tools, and people are trying things. (23:54)

Alexey: You may end up trying this and that and at the end not doing anything. Do you have any suggestions on how we can mitigate this problem? What is the solution when there are too many resources? (24:12)

Learning by Building: Solving Personal Pain Points

Alexey: How do I build a learning path that is for me? (24:40)

Nasser: I would suggest starting with a problem, some real world problem that you want to solve. Maybe it is something that has already been solved, but you are personally interested in solving it. You want to come up with a way to do this that helps you narrow down your focus to a particular domain or a particular slice of AI engineering. (24:45)

Nasser: Quickly start building a solution towards it. I will give you an example. I was on parental leave last summer and one of the big pain points that I have had with going to the gym and working out is tracking my routine and my gym activity. (25:15)

Nasser: I wanted to track things like how many sets I did, how many reps I did, and what the weights were for each of those reps. There are apps that do this, but none of them met exactly what I needed. (25:33)

Alexey: It is funny because if you open the Google Play Store or the Apple App Store, there are thousands and thousands of apps. I have tried these apps and none of them work for me. This is very paradoxical. (25:47)

Alexey: How are we different from all these people for whom these apps are made? Somehow my routine is different in subtle ways that makes it impossible for me to use these apps. I have to come up with a bunch of Excel spreadsheets that I use that are very specific to my routine. (26:11)

Alexey: It is really funny because this is a problem that millions of people have. (26:32)

Nasser: My experience with it was just basic. There were three things I wanted to track: my reps, my sets, and my rest period. I tried so many different apps and none of them let me do it correctly. (26:38)

Nasser: Some of them wanted me to start paying for something or some of them were overengineering for something else. That was a real problem that I had and I just wanted to fix it. During my parental leave, I was just going to fix this. (26:50)

Nasser: I was going to build an app that would do this the way that I wanted to do it. Thankfully we live in a world where it is really easy to build those apps. A side goal here was also to be able to use generative AI technology. (27:02)

Nasser: I wanted to use a parser and use the autocoding assistants to build this and automate as much of this as possible. You will forgive the pun because we are talking about building a gym going app, but one phrase that I frequently share with my team is that if there is something you want to do, you have to get the reps in. You cannot just watch all the courses or read all the books about this. (27:15)

Nasser: You have to actually get out and start building something. The advice I would always tell someone, whether they are an electrical engineer, a sociologist, or a communications major, is to come up with something that annoys you in the real world and try to solve it using these tools. That is the best way to learn because now it is a personal thing. (27:45)

Nasser: Now you have something that is a personal pain point and you want to resolve it and figure out how to adapt the tools for that. When I took that more top down approach, I knew the problem at the top that I wanted to solve. Working towards that helped me narrow my focus to the video I wanted to watch or the capability I wanted to learn. (28:14)

Nasser: This is better than starting from the bottom up where there are so many different videos and side quests you can go on. I would start there. Come up with something that will get you the proverbial reps in the gym to practice. (28:37)

Alexey: That is a really awesome advice. I also give the same. For you, the main thing was to narrow down your focus by focusing on something that matters to you. (28:56)

Alexey: Instead of looking at all the resources and choosing, you come up with a problem and go from there. What are the tools I can use to solve this problem? What are the courses that can help me to solve this particular problem? (29:08)

Alexey: Maybe you do not even need courses? You just need to start building and then at the end you have a working thing and you decide what to do next. You might not even need courses. (29:22)

Alexey: I recently listened to a book where they were talking about an experiment they did. They took people who are artists and put them in a room to create a still picture. They had some pictures on the floor and they needed to paint a picture. (29:42)

Alexey: Some people would just draw. Other people would rearrange things and think about exactly what the best composition was and only then draw. They might not even finish the picture at the end of the experiment. (30:07)

Alexey: What they discovered was that the people who rearrange things became more successful in life later. The book says that there are people who are problem solvers and there are people who are problem finders. The first group was presented with a problem to draw a picture and they just drew it. (30:26)

Alexey: The other ones were thinking they wanted to find a problem and then solve it. They argue that problem finders are more successful in life. I was listening to that and thinking that is cool, but what if I am a problem solver? (30:57)

Alexey: How do I become a problem finder? They do not tell you this. They just say if you are a problem finder, you are more successful. (31:15)

Alexey: I never have a problem with finding problems. They surround me. I have a different problem: how from all the problems I have do I choose the problem I solve? (31:25)

Alexey: For some people I talk to, they have a different problem. They just have their usual work and they do not know what to solve or work on. Your solution is find something that annoys you, but what if it is difficult for me to find something that annoys me? (31:40)

Mental Frameworks for Problem Finders and Solvers

Alexey: Do you have any suggestions for people for whom it is difficult to find problems? How can they find problems? Is there any mental framework you are aware of that can help me narrow down and help me find something that annoys me? (31:50)

Alexey: How do I become a problem finder? (32:14)

Nasser: I love this framework and this mental model of a problem solver versus a problem finder. This reminds me of the common way people think about starting a business. Entrepreneurs think about this frequently. (32:19)

Nasser: There are so many potential problems in the world that I could solve. I do not have hard data on this, but anecdotally from what I have heard and from my conversations with innovators, they start somewhere. That first idea almost never pans out. (32:53)

Nasser: Times when I have started my own initiatives, I have gone in with the mindset that my first idea is going to fail. I want that first idea to fail quickly so that I can get to my next idea and see if that one fails. I keep iterating until I find the idea that sticks. (33:14)

Alexey: Fail fast? (33:27)

Nasser: Exactly. Fail fast. My intuition here is that you pick something and start somewhere that will let you experiment really fast. (33:34)

Nasser: You see if that idea sticks. The worst thing that I can do, and this is something that I encourage my teammates as well, is to have analysis paralysis or think about it too long without making any progress. Coming back to the reps in the gym, just jump in the deep end and start flailing for a while in the water until you figure out what to do or you decide that you want to get into a different pool. (33:47)

Alexey: Is there any way generative AI can help me select a problem to solve? What if I do not go to the gym? There must be something that annoys me. (34:24)

Alexey: I am pretty sure that for everyone, no matter what they do or what kind of responsibilities they have at work or what kind of hobbies they have, there must be something that annoys you. How do I find it? I liked your analogy with business because with business you want to also understand what kind of problems people have. (34:36)

Alexey: I think there is an approach to this problem understanding called design thinking. Have you heard about this? (35:04)

Nasser: Absolutely. Design thinking. (35:10)

Alexey: I have not really worked with product management, but I took a course where they were talking about this. It could be one of the approaches that you could use to find problems in your own life. Maybe just ask ChatGPT? (35:10)

Alexey: Tell it you want to use a design thinking approach. It can be the interviewer and ask questions about your life and help you find a problem that you want to solve. That could be an interesting thing. (35:29)

Nasser: There is a variant of design thinking that is human centered design. Certainly these tools can accelerate that, but I would probably focus more on the human part of human centered design. Maybe the solution is to use Large Language Models and the aggregate creativity of all the textual data they have been trained on. (35:41)

Human-Centered Design in the Age of LLMs

Nasser: An alternative and probably the approach that I would seek is talking to the users and talking to the actual humans. You should make sure that you are capitalizing on the human part of human centered design and figuring out what actually bothers people. If you have got a bunch of things that bother you, find other people that are also bothered by these things and figure out where the critical mass is and where the potential product market fit is. (36:15)

Nasser: This is probably more of a philosophical conversation, which most AI conversations lead into. I would probably discourage using Large Language Models to do the creative thinking that makes us human. Probably encourage people to get out there and get away from the computers to come up with what those problems are. (36:42)

Nasser: Come back to the computers and come back to the agents when you need to actually build a solution. (37:10)

Alexey: We took a little digression and I felt like I spoke more than you. I already interviewed multiple people about the role of AI engineers. Both of them work at a startup and the definition of the role can be very different. (37:21)

Alexey: Both of them told me pretty much the same thing: this is a very full stack role. They get to do everything and at the end even touch the front end. They might not be an expert in front end, but because they have access to all these coding assistants, they do not need to be an expert. (37:51)

Alexey: They just need to know how to instruct their AI assistant to actually do this work and how to apply the engineering rigor to make sure they do not break anything. Even if they do not have a slightest clue about TypeScript, they still know that they need tests. They need certain types of tests like unit tests or integration tests. (38:08)

Alexey: They can apply their engineering background and best practices to solve this in domains they do not know about. You have experience working in more established companies that are not startups where roles could be more defined. In big companies or even midsize companies, you have more specialization. (38:33)

Alexey: In your experience, is the AI engineering role also full stack? If so, is it to the same extent as in a startup or do you see more people specializing in different niches? (39:00)

Nasser: This is a great question. I think it is always beneficial to have a full stack background and to have full stack knowledge and awareness. This does not mean you have to be an expert in all aspects of the stack. (39:14)

Nasser: While I am a strong proponent of using agents and using these code assist tools that are out there, using them responsibly also requires you to have the engineering background and the engineering expertise in those systems. They can be very good ways to accelerate your work. Let us say that someone has a very strong backend expertise and knowledge. (39:46)

Nasser: They are building some sort of app using AI coding assistants and the assistant is building these things using TypeScript or other front end technologies. I think it is always beneficial to have the requisite knowledge to assess the performance of those agents and to assess the quality of the code that is generated long term. You can take shortcuts in a startup where you are really just trying to prove the idea. (40:22)

Nasser: There is more willingness to take on tech debt because it is not as catastrophic for a larger organization. Having a full stack background is always helpful if not necessary in some of the more established companies. I have seen different practices. (41:02)

Nasser: If you are building something that is the fundamental infrastructure for serving AI capabilities or AI agents, then the rigor there might be a little bit different and stronger. If you are building something that you want to prototype out quickly and you want to prove out the concept, then maybe there is a little bit more willingness to let the coding assistants take over and require less oversight. (41:30)

Beyond API Calls: Software Engineering Rigor for Agents

Alexey: We did not really focus on full stack expectations yet. What are the core expectations for an AI engineer? We talked about experimentations and evaluations, but this is just one piece of the puzzle. (42:05)

Alexey: We talked about full stack expectations being helpful but maybe to a smaller extent in bigger organizations. There should be more engineering rigor and one should be more skeptical about introducing tech debt. What are the core skills of AI engineers that are really important? (42:30)

Alexey: With AI assistants, everyone potentially could be a full stack engineer. I am a data scientist and I can change the front end, but everyone is still slightly different. What is the actual focus of AI engineers? (43:01)

Nasser: It is hard to generalize because it is so use case specific. Let us say you are an AI engineer responsible for serving AI agents. It could be anything like a chat app with a deep research component. (43:23)

Alexey: In all these cases, there is a Large Language Model but I am building an agent. In addition to evaluations, what are the other things I need to know? (43:48)

Nasser: It is very dependent on the particular project that you are working on. Certainly if there is a component that is meant to interface with users and you have to collect information from users who do not have a technical background, you will need to have that front end experience. In order to build a system that scales well and is performant, you will need to think about the backend and the infrastructure components. (44:02)

Nasser: A lot of the things that we have thought about in the context of MLOps for a long time are going to matter. How do we make sure that the AI system is continuously improving, continuously training, and continuously deploying? A lot of the automation stuff that we have done around MLOps is going to matter more for AI engineers. (44:37)

Nasser: One other consideration that is different for AI engineers compared to other adjacent engineering roles is the orchestration element. So much of building agents requires orchestrating decisions from one tool or one agent to sub agents. You have to think about how those decisions are made. (45:02)

Nasser: For an agent call or an agent invocation, there are multiple aspects of the orchestration to consider. Also for the evaluation component, how do you make sure that you are taking the outputs of that evaluation and then feeding that back into improvements or changes in your agent? (45:35)

Orchestration and the Rise of Agent Ops

Alexey: Orchestration is very important. The agent needs to decide to invoke this tool, so we need to be able to make this possible for the agent. In your experience, is it really important to fixate on the tool and learn this tool well? (45:50)

Alexey: Or is it more important to learn concepts like evaluation or orchestration? (46:32)

Nasser: This is a good question. I think it depends on your stage. If you are learning, I would start with one of the known tools. (46:40)

Nasser: There are certainly not that many that are top contenders. I would start with one and focus on developing depth of understanding in that one before trying to experiment with different ones and seeing their performance against each other. As you do this more and coming back to the gym analogy, if you are training a single muscle, you do not need to try all fifteen exercises that will train that muscle. (46:56)

Nasser: You start with one. Gradually as you either start to see diminishing returns or you want to further build that muscle, you start to look at the other agents or tools and assess them against each other. I imagine increasingly we will see in orchestrations, particularly in startups, some cross fertilization using different tools and frameworks. (47:34)

Nasser: Companies will try to figure out which works well for a use case even if that means handing off the next step of this to a completely different large language model or framework. (48:12)

Alexey: I like your gym analogy. Maybe because this is something I can relate to. I know that there is this framework called five by five which are the top five exercises that get you the entire thing. (48:24)

Alexey: Is there a five by five analogy in AI engineering? What are these top five exercises that get you a full body experience? (48:38)

Nasser: I would have to think about that one. You can be biased. ADK is the one that I have the most experience with and it is the one that I use more on a day to day basis. (48:55)

Nasser: I would encourage anyone to experiment with all of them and try to figure out which one works best for them. In the beginning, just start with one. Whether it is a tutorial that you are going through or a YouTube video that you are watching, start with something and test it out. (49:15)

Nasser: Try to get the experience in yourself and build some depth and breadth there before branching out to another one. (49:40)

Alexey: I mentioned this five by five and one of these exercises is a deadlift, which can be pretty dangerous for novices. The analogy I had is these graph agentic orchestrators. I guess this is not something we should start with when we just start learning about agents because these could be very difficult things to handle. (49:46)

Alexey: When I open Twitter, I see people talking about LangGraph and other things and it feels like I need to drop everything and start learning this thing. (50:12)

Nasser: I am generally of the opinion that these things are very much use case dependent. I would want to try to experiment with all of them, but focus on the ones where other people who have built similar things have had success. What frameworks did they use? (50:25)

Nasser: What things were helpful for them? Maybe do not spend too much time going too far down the rabbit hole of one of them before you can go back and experiment with something else. It is hard to say that this is the one you should definitely use or definitely avoid. (50:56)

Alexey: Probably what you are trying to say is just choose any and go with it? (51:21)

Nasser: I would personally recommend that. It depends on the organization that you are working with or if there are any limitations. If you are working in an organization that is primarily using tools from one vendor, then focus on those. (51:21)

Depth vs Breadth in AI Framework Selection

Alexey: ADK is cool and I like it too, even though usually my go to framework is a different one. I do not think it matters in the end. Just choose any. (51:30)

Alexey: I see another question. How do you see the AI engineering role evolving over the next years? I know you do not have a fortune ball, but you have been in this industry for long enough to see where it might be going. (52:00)

Alexey: Do you think it is going to be more engineering heavy or less engineering heavy? (52:31)

Nasser: I think we will see more engineering rigor as ADK introduced. Data science saw this as well. For a long time, a lot of what we called data science was happening in tools like R and sort of not fully baked libraries in Python. (52:44)

Nasser: Then Scikit-learn came, Keras came, and all these abstractions came. I think we will see the same thing happening with AI engineering. I also see more data science elements coming back. (53:10)

Nasser: We talked about this a little bit around the way that data science emerged into machine learning engineering, which emerged into MLOps. We focused more on things like monitoring the performance of these agents using concepts like data drift and concept drift. I think we will see this increasingly in AI engineering. (53:22)

Nasser: This field of agent ops will continue to evolve. (54:03)

Alexey: Agent ops? Is that a thing already? I have not heard this before. (54:08)

Nasser: I feel like I have heard it in a couple of spaces. I know certainly at Google we are taking a look. (54:08)

Alexey: There is an agentops.ai website. It is nice to have another ops thing in our collection. (54:21)

Nasser: The point is that in some of my own projects, I see that we are thinking more about the not just the engine. We oscillate. AI engineering is largely a statistical data science exercise, then it becomes more of an engineering exercise. (54:33)

Nasser: I think it will again continue to become a statistical data science exercise as well. One other hope that I have is that Large Language Models and agents in general become faster. In the absence of them becoming faster, a key advantage of data science and traditional machine learning was the speed of inference. (54:59)

Nasser: Autonomous cars are a good example. Autonomous cars cannot run purely on agents just for the sake of latency. We cannot run sonar or lidar signals through an agent and say there is a human there, and then wait for a few seconds before we actually react. (55:30)

Nasser: There needs to be a much faster response. I can imagine a lot of what we build in the context of using Large Language Models and generative AI, we will start to discover patterns. We will ask if there is a way to turn this into a structured data science exercise and turn this back into a very low latency traditional machine learning model. (55:47)

The Future of Latency and Traditional ML Integration

Alexey: For example, if I am doing search, I can use a Large Language Model right now but it is very slow. What if I can use the data that the model generates in order to train an XGBoost model that is fast? Then I deploy the model instead of that. It could be even faster than the original model because Large Language Models are generalists and not specialists. (56:10)

Alexey: Have you heard about these small language models? (56:49)

Nasser: Yes. It is one of the trends and that solves a lot of the latency problem too. I will answer your question about where AI engineering is going to go in a different way. (56:55)

Nasser: As the field was emerging, my vision was always that using these higher latency models would be a way to quickly prototype and solve problems in a fast way when we do not have the time to iterate on coming up with structured data sets. I always thought of this as an intermediate step. At some point we will want this to be faster. (57:19)

Nasser: If we are using it to solve traditional machine learning problems because we have too much unstructured data or do not have the time to clean it, we will at some point want to convert that into something that is lower latency. Ultimately, traditional machine learning models would still be the goal. The field evolved in a different direction where we went from these single Large Language Model requests to these agents that orchestrate multiple requests and tools. (57:49)

Nasser: I can still see that at some point we will say that we have learned enough from building these agent solutions. Is there a way that we can make this faster using traditional machine learning? Maybe we will see data science come back. (58:30)

Alexey: Do you have a few more minutes for the last question? (58:54)

Nasser: Yes, I believe so. (59:01)

Alexey: The last question, I promise. We talked about these small language models, distillation, and maybe people also heard about fine tuning. All these things sometimes pop up and this creates another inability to decide what to do next because there are so many things. (59:01)

Alexey: If we follow this route of fine tuning or understanding how models work or self attention, how important is it to know these things if I am just getting started? If I work as an electrical engineer and I want to get into this AI agents world, how important is it for me to understand all these things that are more theoretical in order to start building? (59:32)

Nasser: These are certainly more than theory now. There is work being done and it is very much practice to fine tune and to distill models. It is helpful to know about these things at an abstract level when you are getting started. (59:54)

Nasser: If you are building your gym tracking app, maybe it is overkill in the beginning. Over time as you start to productionize what you are building or you want to see better improvement or lower latency, maybe you want smaller models that can run on your phone. Then it is worth digging into these things a little bit more. (1:00:25)

Nasser: While you are learning and things like latency or the device do not matter as much, a theoretical awareness of it is helpful. (1:00:56)

Alexey: Once you decide that it is time to deploy your model to your self driving car, then maybe you need to learn about distillation? (1:01:02)

When to Prioritize Model Distillation and Fine-Tuning

Nasser: Absolutely. Once you have got a big enough use case for those things, but certainly as you are learning, it is just helpful to know that they exist and that they are capabilities. (1:01:20)

Alexey: Just to be clear, I do not recommend using Large Language Models for self driving cars. Thanks a lot. I really enjoyed this conversation. (1:01:26)

Alexey: I know I took a bit of a turn from the prepared questions, but I really enjoyed this conversation. Thanks a lot for joining as a guest and thanks everyone for joining us today. That is all we have time for today. Anything you want to say before we go? (1:01:40)

Nasser: No, I appreciate the conversation and the questions. I am looking forward to learning more. (1:01:57)

Alexey: Thanks. Have a great day and bye everyone. (1:02:03)

Nasser: Bye all. (1:02:09)