Building an AI Platform: SDK Design, Model Routing, Vector Search & Evals | Suhas Suresha

The transcripts are edited for clarity, sometimes with AI. If you notice any incorrect information, let us know.

AI Prototypes to Production and the Book Origin

Alexey: Hi everyone. Let us get started. I am going to skip our usual introduction but as always please like the video and subscribe to our YouTube channel. There is also a link for our Slack community where you can hang out with other data enthusiasts. It is actually almost 100,000 people. (0:00)

Alexey: I am really wondering who is going to be the lucky number. Right now we have four to go. Perhaps we have your books. Is it true that the person who is going to be number 100,000 is going to get a free copy of the book? (0:20)

Alexey: We can make it happen because it is really amazing. If you are not a part of our Slack go join it. For today's interview there is a pinned link in the live chat. Click on that link to ask your questions. We will be covering these questions during the interview. (0:39)

Alexey: That is the usual introduction. Usually I use slides for that. Now we will start with our interview. (1:01)

Alexey: Today we are going to talk about building an AI platform. We will talk about how to go from a successful prototype to something that serves real customers. We have a special guest today named Suhas. He is a senior machine learning engineer at Adobe where he builds large scale generative AI platforms. The main reason we are here today is the book that he and his coauthor wrote recently. (1:17)

Alexey: It is still in early access but it is going to be published pretty soon. All the content of the book is already out. All 10 chapters are available to read. How long did it take? (1:47)

Suhas: We kind of did it pretty fast to be honest. It took us maybe a year and a half. (2:05)

Alexey: Do you call a year and a half fast compared to what I have heard? Other folks write books slower I think. (2:13)

Suhas: The bulk of our time we probably spent six months. Six months is what it really took. There is also this production preproduction postproduction and during production. (2:24)

Alexey: Manning has a very rigid process. I know this because I also have a book with Manning. It took two years. (2:37)

Suhas: The content was not the difficult part. (2:43)

Alexey: Going through all the reviews and production takes time. At the end you want to have a book that is good quality do you not? These processes are in place to ensure that the book when it is published is great. (2:49)

Suhas: That is exactly right. (3:03)

Alexey: The book is called Designing AI Systems is it not? Tell us more about that. (3:09)

Suhas: The inspiration for the book came about maybe a year and a half or two years ago. My coauthor and I were seeing that there were a lot of books on how to build really cool generative AI applications. People would have a lot of blog posts and books about it. When we had to deploy these we also built proof of concepts. We showed them to our bosses and everyone was super happy. (3:15)

Suhas: When we realized we had to take these to production it was so much more work than that. We felt that we only spent maybe five percent or even two percent of the time actually working on the generative AI stuff. A lot of time was spent on so many engineering tasks. We felt this is actually not super easy to get out into production. (3:48)

Suhas: You can get something out there into production but it is hard to ensure it works well and has a high accuracy. It needs to be able to maintain state across conversations. It also needs to be able to ground its answers in real documents or real data. The system must use tools safely and have the right guardrails to scale well. We need to be able to observe how much we are spending on all these different AI models. (4:19)

Suhas: Having that observability is crucial because we realized all these features are not just nice to have. They really become musthaves. (5:00)

Alexey: It takes work to take something working on your laptop as a proof of concept to actually serving users and doing well. Especially now when you started the coding agents were not that good. Now you can have a working prototype in one evening can you not? Is this prototype good enough for production? (5:08)

Suhas: All of these things do get faster with coding agents. Even then sometimes folks do not even realize what needs to be done. That is when they fall into the trap of not estimating the time properly to get something out. That was one aspect of taking something from a demo to production. Where does the platform angle come in from? (5:28)

AI Sprawl and the Need for a Centralized AI Platform

Suhas: The subtitle of our book is actually a guide to building production ready platforms. That is a slightly adjacent thing but a very important thing as well. If you are just a startup of one or two people and you only have one or two AI applications the platform itself is not necessarily going to speak to you that much. (5:49)

Suhas: If you are a decent size startup where you have at least five to ten different AI projects going around a platform becomes pretty valuable. Without it every team has to build all these other aspects I just mentioned. This includes building session storage and vector search. They also have to ensure they have MCP integration and that tool calls are done in a secure manner. Adding guardrails and other features becomes a lot of duplicated effort. (6:28)

Suhas: If you are a leader in a company and you want to observe what is happening across a lot of AI applications it becomes very difficult if you have fragmented efforts. We call this AI sprawl in our book. (7:09)

Alexey: What is AI sprawl and what does sprawl mean? (7:28)

Suhas: Sprawl refers to things that organically happen across the organization. It looks fine initially but then you have too many things to consolidate together. (7:35)

Suhas: I will give you a simple example regarding observability of AI systems. Let us say five different applications build observability in five different ways. The data model for observability is vastly different. One day the CTO wants to know how things are looking and someone has to reconcile all these different models to build a common observing board. That becomes a new block of work but the worst part is the consolidation. (7:46)

Suhas: Whenever someone starts a new AI project they do not have anything to start off with. They have to start again which leads to more and more sprawl. Let us stop the sad stories and think of the ideal scenario. The magic of generative AI is that you get something really cool out in a day. Within a few minutes it is in production and you automatically get a lot of these things for free. (8:22)

Suhas: That is the idea. A lot of the big tech companies do this really well. They have really good platforms which is why they are able to move really fast. I will give you an example. Take Perplexity for example. (9:02)

Alexey: Are you referring to the platform? (9:21)

Suhas: I mean the Perplexity app as well as the platform. (9:27)

Alexey: I still have not used it but I understand what it is. It is kind of RAG on a very large scale is it not? (9:35)

Suhas: Take Cursor or another coding agent you have used. (9:41)

Alexey: I use all of them except Cursor. (9:46)

Suhas: I am trying to think of something that is a good example. (9:52)

Alexey: Whenever a new model comes out Anthropic announces it at ten o'clock and by ten five these folks have it. They have that model available immediately. I am pretty sure they agree in advance so they just delay their announcement after the official one. GitHub Copilot speaks with Anthropic so Anthropic tells them there is a new model. (9:57)

Suhas: That is true but the way they have built their system makes it very easy for them to swap out models. When they build their model service they ensure it is very easy to switch providers or models. When you are starting to build out an application as a proof of concept you do not think of all that. You do not think of what happens if you just use Anthropic because you are very focused on your idea. (10:22)

Alexey: Yes exactly. (10:56)

Suhas: That is the origin of the book to be honest. We saw that nobody else was writing about the ninety eight percent of the work that needs to be done to take something to production. We thought let us write a book. (11:02)

Alexey: It is a very good idea. While we were talking I looked up the word sprawl. In urban development it means the uncontrolled expansion of a city into low density automobile dependent surrounding countryside which increases traffic pollution and infrastructure costs. When I was reading this definition I imagined an organization where many teams start creating agents. In order to deploy these agents we need an evaluation harness. (11:14)

Alexey: Each team now has its own evaluation harness. To speed up rolling out to production faster they start developing a platform. At the end you have fifteen or twenty teams that have their own thing. As a principal engineer you wonder what to do with this disconnected infrastructure. (11:55)

Alexey: Some things are suboptimal here and some things are suboptimal there. You want to bring some sort of economy of scale by centralizing it in one place. This makes it easier for teams who do not necessarily have this platform yet to also roll this out. This is the idea is it not? (12:24)

Suhas: That is exactly right. (12:35)

Definition of an AI Platform and the Build vs Buy Dilemma

Suhas: Once you have the platform you see all the gains. Our next discussion is how you build such a platform. The most common question we get is why someone should learn to build a new platform from scratch. There is actually a lot of really cool platform stuff out there like LlamaIndex. (12:42)

Alexey: Is it a platform though? For me LlamaIndex is a library for doing RAG although they offer hosted solutions. Maybe what we can do first is define what a platform is before we decide why we should learn to build one. (13:29)

Suhas: That is a good question. The way I define a platform is something that developers use to build their applications on top of. A good platform should ideally give you a lot of value for free. A good generative AI platform should not require developers to worry about most production concerns. (13:57)

Suhas: They should not worry about how to scale their application or connect to a session store or a vector database. Most of these things should be taken care of by the platform. The only thing the developer is worried about is adding their business logic. They should only be able to focus on delivering their specific application really well. (14:32)

Alexey: I do not know if you have used Vercel but I was amazed how easy it is. I am not affiliated with them but I tried it and it is so easy to create an agent there. You define a markdown document with instructions and another file with the definitions of your tools. That is basically your entire agent. With one command it gets deployed to a serverless environment. (15:07)

Alexey: It gets connected automatically to this AI gateway or whatever they call it. It scales and you get observability and durability out of the box. I was amazed by how cool it is. This is an example of a platform is it not? (15:37)

Suhas: Absolutely. You gave the best example. Vercel is fantastic and all the things you mentioned are exactly what we write about in the book. We explain how you build such a platform to get this magical experience you just spoke about. You are just worried about writing your application or business logic. (15:57)

Suhas: You get everything else like an agent URL and observability automatically. You can define how scalable you want this application to be. All of this is essentially just knobs that you can control. You do not have to build any of it. (16:22)

Alexey: If Vercel exists why do we need to even care about building a platform? (16:46)

Suhas: That is a great question. You can use Vercel and it is totally fine. (16:52)

Alexey: Except you have to tolerate JavaScript. (16:57)

Suhas: There are limitations. I have not used Vercel extensively so I do not know all the limitations. It is very TypeScript and JavaScript focused. If you want to get off the main track you have problems. With Python it kind of works but it is not the same experience. (17:00)

Suhas: That platform may not be applicable for exactly what you want to do. We felt that most of the time the right solution is to use an existing platform. I would not advise someone to completely build a platform from scratch especially if they are a solo founder. (17:25)

Alexey: Exactly. Do not do that. (17:45)

Suhas: Learning how to build a platform teaches you a lot of things. It teaches you what is happening behind the scenes. People like me always love to know what is happening behind the scenes. (17:52)

Alexey: That is useful because it also gives you a good understanding of which platform to choose. If you are choosing between two or three different vendors for a platform you get an idea of what you are getting from them. You understand it because you have essentially already built one. (18:03)

Suhas: If you are joining a big tech company knowing this is very important. Many midsize to big tech companies have very specific applications that they need to build. Most of them build their own platforms. You might actually be hired to work on the platform. In those scenarios it becomes very important for you to understand how to build a platform from scratch. (18:23)

Alexey: I remember my days working as a data scientist where we had the same story with machine learning platforms. Many companies were building their own platforms. I wondered why they would do this if there are platforms like SageMaker or Databricks. Why on earth would anyone start building their own platform when there are many providers available? The problem is integration. (19:02)

Alexey: You already have existing systems and you want a solution that integrates well with your internal ecosystem. Sometimes it is much easier to develop something inhouse that integrates perfectly. If you pay a vendor you often spend the same amount of time integrating their product as you would spend writing your own thing. When you build your own thing you know exactly how it works. (19:27)

Alexey: Of course it is a journey and it will not work perfectly on the first attempt. When I realized how much integration is required I concluded these teams know what they are doing. Sometimes there is the not invented here syndrome but I completely understand the technical need. (19:57)

Suhas: These are all tradeoffs. Sometimes you have the risk of vendor lockin and you cannot easily migrate. Vercel is great but if you start scaling it can become very expensive. It is very cheap if you have one customer but with one million customers it suddenly becomes costly. (20:20)

Alexey: Exactly. The same thing applies to lower level agentic frameworks. From what I see practitioners end up implementing their own frameworks. I often wonder why they cannot just use Pydantic AI. The reality is their requirements are so specific that they need to stop tool calls at an exact moment. (20:45)

Alexey: For them it is just easier to implement their own agentic framework rather than fighting LangChain or Pydantic AI to get what they need. The same applies to platforms does it not? (21:09)

Suhas: Absolutely. Even with the platforms we write about in the book we do not build everything from scratch. For example when we discuss the data service we do not build a vector store from scratch. We show an example of how to build one using Postgres. The idea is you can build a platform in such a way that you can easily swap in different providers. (21:23)

Suhas: This is commonly done in the industry. I have seen many platforms where you are building an application and need to ingest data into a vector store. The platform gives you options between Postgres or other databases. A platform does not require you to build everything from scratch. You can have different vendors plug in at different points of the platform. (22:05)

Alexey: You have to have this plug and play architecture. If you are not satisfied with Postgres and want to switch to Elasticsearch you just replace that component. The rest of the whole system remains the same. (22:47)

Suhas: Exactly. Or if you no longer prefer Anthropic you can switch to a different LLM. (23:04)

Alexey: For you it is just swapping one component. I understand. (23:10)

Suhas: That is correct. (23:15)

AI Platform Engineering Skills and Data Service Architecture

Alexey: I understand that having a platform is good. We have a specific request from people in our community. Sometimes people write to me saying AI engineering seems interesting but they are more interested in AI platform engineering. They want to know what they need to do to work as an AI platform engineer. My suggestion is that you first need to understand what AI engineers do and then work backwards from there. (23:17)

Alexey: Since I do not have direct experience with platform engineering I want to ask you. What should be the path for a backend engineer who is interested in AI but prefers platform engineering? How do they transition to being an AI platform engineer and what skills do they need to acquire? (23:51)

Suhas: That is a great question. A lot of the book was written from exactly that point of view. Another question we get asked often is why someone should learn to build an AI platform when it seems similar to building any general software platform. Most of the core software concerns seem to be the same. That is where we identify what is completely new with generative AI and highlight the specific areas you need to master. (24:27)

Suhas: Let us say you are a backend engineer currently working on traditional software development and you want to shift to AI platform engineering. It is very important for you to first understand how these agents work. Think of the lifecycle request of an agent and what happens when a user sends a request. I know agents are wildly different in terms of harnesses but many of the underlying steps are similar. (25:03)

Alexey: Things like tool call loops are under the hood of any framework. (25:39)

Suhas: The agentic loop is a pretty standard concept. Understanding those fundamentals becomes very important. Once you understand that you can work backwards and identify all the services you need to build for a platform. We agreed initially that the platform should ensure developers do not have to worry about infrastructure aspects. (25:50)

Suhas: This includes scaling and connecting to a vector database or a model service. It also handles invoking services to enforce guardrails. All of these things should not be developer concerns. (26:30)

Suhas: These infrastructure components should be built by the platform. You should start by understanding how an agentic request flow works. Then you can work backwards to map out all the different services you need to build. For example you need to build a model service to ensure developers are free to choose between different model providers. (26:50)

Alexey: Something like LiteLLM or a similar router. (27:27)

Suhas: You will understand that just routing is not enough. What happens if a request fails from a particular provider? You need to handle failovers to ensure the system does not abruptly fail. The platform needs to seamlessly switch to another provider as part of the request flow and manage rate limits. There are many operational concerns to address when you want to take an application to production. (27:34)

Suhas: Let us take the data service as an example. You need to understand how vector search works end to end. When you see tutorials on vector search people usually just explain the basic concept. I will tell you what most people actually end up spending their time on. A major part of vector search involves adding a step to identify the format of the ingested documents. (28:12)

Suhas: These documents may be PDFs or Word files or Excel sheets. Identifying the format and adding specific parsers to extract the content takes a lot of time. There are a lot of nuances involved in parsing. Extracting content from a PDF is notoriously difficult because they contain complex layouts like tables. (28:54)

Suhas: Sometimes all the text is in one block and other times it is split across paragraphs. It is incredibly difficult to extract text and ensure that it reads correctly. (29:26)

Suhas: Then you have to manage chunking strategies. You have to ensure you provide options so developers can mix and match different chunking strategies for their application. If they want to add their own specific logic you must provide a way for them to register a new chunking strategy. The platform also needs an interface where you can plug in any vector database like Pinecone or Elasticsearch or Postgres. (29:43)

Suhas: The ultimate goal is building an interface where the developer only writes one or two lines of code. They just pass the user query to the vector search component. The result is a fully prepared tool that the agent can call immediately. (30:26)

Alexey: It can be registered as a tool in some applications. If you know you are doing RAG every time it becomes a mandatory step that happens even without the model explicitly invoking that tool. How you build it is very application specific. The platform should essentially enable developers to do either of these approaches seamlessly. (30:52)

Suhas: Let us think of AI observability next. (31:17)

Alexey: I want to do a quick recap first. We are trying to understand the core abstractions needed for a generative AI application. The first thing is the model service which handles failures and rate limiting and provider switching. That acts as our router. The second component is the data service which handles data ingestion and makes context easily available for your application as a tool or a mandatory RAG step. (31:26)

System Observability and Request Flow Bottlenecks

Alexey: We spent so much time discussing just those two components. Now you are touching on observability which is the third component. (32:26)

Suhas: The model service is a completely new concept for these applications. Vector search is not a new concept but many new nuances have come into the picture. You have to ensure that the search results are well integrated into the final prompt. You cannot stop at just retrieving the relevant documents. You also have to understand how to format and insert those chunks correctly into the context window of the LLM. (32:34)

Suhas: With observability most of the concepts are the same as traditional software. In a distributed system you want to understand how much time each component spends during a request lifecycle. For example consider a RAG flow where a user sends a request. You send that query to the data service which performs a vector search to fetch the relevant documents. You then add those documents into your model context and make a call to the LLM. (33:20)

Suhas: The LLM processes the user question alongside your system prompt and the retrieved grounding context. It uses this information to generate an answer grounded in reality. In this case you need to know exactly how much time was spent in the data service and the model service. Let us say you also add guardrails to ensure you do not leak personally identifiable information or company secrets. You might use a smaller model to check the final answer and you need to track how much time that security check takes. (34:07)

Suhas: You want a complete view of the time spent across these different components for every request. You need this data as an aggregate across all requests. If your boss asks you to optimize latency from an average of two seconds to under one second you can pinpoint exactly where to optimize. If the bottleneck is the model service you might look at using a smaller LLM. (35:05)

Alexey: You can clearly identify the bottleneck because RAG steps are strictly sequential. You can see if you spent one second in one place and twenty milliseconds in another. If the data service is the bottleneck you know exactly where to focus your optimization efforts. (35:47)

Suhas: That is exactly right. These observability tools become even more important with agentic loops. Coding agents can go into a loop for thirty minutes continuously executing tool calls until they are satisfied with the answer. (36:01)

Suhas: They do not stop and will keep calling as many tools as necessary. Observability lets you analyze how many times a specific tool was invoked during a single query. You can identify which tools are taking the most time to execute. This data feeds directly back into your optimization strategy. If a useful tool takes a minute to execute and the model calls it twenty times you clearly need to optimize it. (36:19)

Suhas: You might rewrite the tool description or change how it performs completely. Observability becomes extremely key in generative AI systems. That is why understanding the request flow of an actual agent is so important. Once you understand the flow you know exactly what infrastructure you need to build. (37:00)

LLM Evaluations, Prompt Experimentation, and Cost Tracking

Alexey: There are a couple of questions from the audience that I will get back to later. I want to finish mapping out the essential pieces of the platform first. (37:32)

Alexey: We want a complete understanding of what components this platform requires. So far we have covered the model service and data service and guardrails and observability. Observability allows you to trace the request flow and understand exactly how much time is spent at each step. What else is important to include in our architecture? I think we previously mentioned evaluation and durability. (37:45)

Suhas: Let us discuss evaluation next because it is closely tied to observability. Typical software systems are very deterministic so your failure rates are based on standard HTTP status codes. As long as you get a successful response code you assume the service is functioning well. That is not true for AI applications because you might deploy a chatbot that is totally useless. It might be working perfectly from a traditional software engineering perspective with zero failure rates. (38:21)

Alexey: It just constantly recommends the user go read the documentation instead of actually helping. (39:01)

Suhas: Exactly. (39:07)

Suhas: If it does not actually help the user then it cannot be considered a success. That is where AI evaluations become critical. It is very important to sample a percentage of your production requests. You should evaluate those samples using either automated or human scorers. (39:07)

Alexey: Because we have observability we already collect these system traces. Once we have these traces we can run them through a system to determine if we actually achieved the goal of the request. We can evaluate if the model genuinely helped the user. We can sample thousands of logs to get an accurate idea of how good the system actually is. (39:38)

Suhas: Exactly. (40:07)

Suhas: That data feeds directly back into experimentation because evaluation and experimentation are fundamentally linked. You analyze the failure data to create a feedback loop for whatever you need to change. If you feel the model is underperforming you might switch to a different provider. You might decide to completely change your orchestration harness. (40:14)

Suhas: For example you might discover that running the data service before the model even calls any tools improves performance. You can clearly see the tradeoffs because adding certain steps might increase latency while boosting user satisfaction. The final angle to consider here is operational cost. (40:46)

Alexey: Cost tracking is tied directly into the observability platform. This is where we measure expenses because the majority of the cost happens in the model service. (41:18)

Suhas: There is also cost associated with vector search operations. Many developers use local embedding models where they only pay for raw compute power. A lot of companies also offer managed embedding models as a service. (41:30)

Alexey: Cloudflare has a service called Vectorize where you pay per request. It is very convenient but eventually at scale you are paying for every transaction instead of just compute. Different guardrails also contribute to the overall cost so you have to sum everything up. (41:50)

Suhas: You have to aggregate all those expenses and add them to your tradeoff matrix. If you optimize entirely for cost you have to make sacrifices in performance or latency. (42:15)

Alexey: I want to speed things up a bit to ensure we have time for questions. You mentioned experimentation as another core component of the platform. We need tools to compare Model A and Model B in production. What else does the platform need? (42:34)

Suhas: I would not just call it Model A and Model B because the model itself is rarely the only variable. These days the variables include the model and the system prompt and the tool definitions. (42:52)

Alexey: We should call it Version A and Version B. (43:04)

Suhas: We use the term artifact to describe the complete combination of the harness and tools. (43:10)

Alexey: It includes all the orchestration logic plus the model itself. (43:17)

Session Memory, Tool Registries, and Agent Orchestration

Suhas: You should treat that complete bundle as versioned code. It is critical to know that a specific combination of the harness and prompt yields a specific performance metric. When you introduce variations you can version them and trace the exact impact on accuracy and latency. The platform should provide this infrastructure so teams can experiment rapidly without relying on local files. (43:57)

Alexey: What else should we have in the platform? (44:04)

Suhas: The next important component is session and memory management. (44:09)

Alexey: One of the audience questions actually asks how to manage short term and long term memory for the application. This memory management is exactly what we are discussing. Why do we even need it? (44:15)

Suhas: The simplest explanation is that large language models are completely stateless. Every time you send a request you have to pass in all the context required for the model to answer correctly. The model does not retain memory on its own so you have to build that infrastructure. Think of standard chat applications where you maintain multiple active sessions simultaneously. The simplest requirement is ensuring the model remembers previous turns within a single conversation session. (44:33)

Alexey: That handles the short term memory requirements. (45:21)

Suhas: The second requirement is building a cross session understanding of the user. Developers often implement a RAG pipeline across a user's historical sessions. You see this when you ask a question and the system searches across all your previous chats. The platform is continuously indexing all the various conversations you have had. When you ask a new query it performs a vector search on that index to fetch relevant historical context. (45:28)

Alexey: Sometimes when I use coding agents to write documentation they spontaneously decide to write tests for that documentation. I have to correct them and tell them tests are not needed. The agent then creates a long term memory stating it should not write tests for documentation and saves it for the project. (46:11)

Suhas: That is exactly the third concept I was getting to. (46:36)

Suhas: We call this concept model managed memory. You offer long term memory as a standard tool and the model itself decides when to invoke it. This saved memory is then automatically appended to the context window on every subsequent turn. (46:41)

Alexey: You obviously still need traditional session management to run cross session searches. What else do we need in the platform? (47:00)

Suhas: We absolutely need a dedicated tool service. Agents and tools go hand in hand because the execution loop is what actually defines an agent. You need a centralized tool registry and automatic tool discovery. The platform must also guarantee the safe execution of these registered tools. (47:11)

Suhas: There are many new protocols available today like the Model Context Protocol. You have to ensure that developers can easily add MCP tools to the platform. A dedicated tool service becomes incredibly important for managing all these integrations. (47:44)

Alexey: Tool usage is another direct contribution to the overall cost. If we use an external MCP server that costs money we need to track it through the observability pipeline. (48:10)

Suhas: We already covered guardrails briefly earlier in the discussion. Developers must be able to write custom guardrail policies that the platform automatically enforces. (48:17)

Alexey: Should the orchestration harness itself be a core part of the platform? (48:36)

Alexey: What exactly is a harness? Is it the same as a framework like LangChain? (48:44)

Suhas: For an agent the harness is anything built outside the core model call. Everything wrapping the model is basically the orchestration harness that makes the agent useful. The standard agentic loop is a perfect example of a harness. Not every application actually requires an open ended tool loop. (48:51)

Suhas: There are tons of applications that use a simple directed acyclic graph approach. You know exactly what sequential steps the AI needs to execute so you just fix those steps in place. That fixed pipeline becomes a very simple harness. (49:17)

Alexey: The platform should let developers build these distinct workflows seamlessly. Automation tools like n8n let you define a strict sequence of execution steps. That fixed sequence represents the DAG approach. This is fundamentally different from giving an agent access to tools and letting it decide what to call. (49:36)

Suhas: There are many different ways of building orchestration. In our book we demonstrate writing a custom agentic loop in just fifty lines of Python code. What makes that simple harness scalable is the underlying platform infrastructure. When the code calls the model service the platform completely abstracts away the complexity. (50:00)

Suhas: It automatically handles provider failovers and dynamic request routing invisibly. You can write and optimize the core harness logic yourself. You still get the full power of the platform handling the heavy infrastructure lifting for free. (50:25)

Invocation Modes and Durable Execution for AI Agents

Alexey: That explains why Vercel lets you define your prompt as a markdown document and your tools as JavaScript files. The complex orchestration is already built directly into the platform. (50:56)

Suhas: The platform also needs to offer different invocation modes for the deployed agents. (51:02)

Suhas: We define three distinct invocation modes for different use cases. Traditional software applications execute processing and return the final result synchronously. That synchronous mode is still valuable for fast classification tasks. If you are building a chatbot you cannot force the user to wait for the entire response to generate. The platform absolutely needs to support native streaming outputs. (51:09)

Suhas: The third mode is asynchronous execution which is crucial for deep research tasks. If you use advanced models for deep research they do not return an answer immediately. The system tells you to check back in ten minutes while it works in the background. The user application simply polls the server to check the status of the long running task. The agent handles whatever complex execution is required to generate a satisfactory answer in the background. (51:50)

Alexey: You mentioned three modes so the first one is streaming and the second is asynchronous polling. Are those the three modes? (52:38)

Suhas: The three modes are synchronous and asynchronous and streaming. (52:49)

Alexey: By synchronous you mean standard request and response architecture. (52:54)

Suhas: With synchronous execution the user sends a request and waits for the agent to compute the full answer before returning it. With streaming the user progressively receives tokens without waiting for the complete generation. In asynchronous mode the user makes a request and immediately receives a polling URL. The system tells the client to keep checking that URL while the agent works in the background. (52:54)

Alexey: I think there is also a fourth one like a batch execution mode. (53:42)

Suhas: Batch execution is crucial for running massive evaluations or processing complex document ingestion pipelines. (53:49)

Alexey: Where does durable execution belong in this architecture? Is it a completely separate component? (54:03)

Suhas: Durable execution is something that needs to be built directly into the orchestrator. (54:18)

Alexey: Durable execution is the ability to pause and continue execution after an interruption. If you need approval from the user or a data center catches on fire you want the execution to safely resume elsewhere. I assume that functionality is fundamentally tied to session management and orchestration. (54:30)

Suhas: Durability should actually be built into all the services across the platform. If something fails at the model service layer you need automatic retries and failovers managed there. If a failure happens at the orchestration layer the gateway itself needs the ability to safely retry the user request. The reliability of a platform is not localized to just one single stage. It is a core architectural requirement that spans all the services we are discussing. (54:55)

Alexey: I have listed the model service and data service and observability and guardrails and evaluation. I also have experimentation and session management and the tool service and the orchestrator. Is there something missing from this architecture? (55:46)

Suhas: That is a very comprehensive list of nine distinct components. You can build ninety nine percent of your applications using just these systems. (56:08)

The Nine Core AI Platform Components and Memory Management

Alexey: A simple proof of concept is just defining the prompt and the tool calls in one evening. All these other infrastructure components still need to happen before production deployment. A truly good platform abstracts all this complexity away entirely. It allows us to take an application developed in one evening straight to production with minimal friction. That is the ultimate goal of building this platform. (56:14)

Suhas: That is exactly right. (56:58)

Alexey: We need to wrap up soon but I want to answer an audience question regarding memory management. The question asks if memory is mandatory for all agents and why it should be treated as a separate architectural service. (57:03)

Suhas: If you are running a coding agent locally you can simply save memory as local files on your laptop. If you are building an enterprise application serving tens of thousands of users from remote servers you need a robust persistent store. You have to connect your agents to a production database like Postgres or MongoDB to store those conversations securely. (57:35)

Suhas: The application developer needs a standardized abstraction to read and write to those databases seamlessly. Handling persistent state safely across thousands of concurrent users is a major production concern. That is exactly why you need a dedicated session service handling the infrastructure layer. (58:19)

Alexey: I can add that some applications completely lack the need for memory. It is highly use case dependent so I would not consider it a mandatory architectural step for every single agent. (58:52)

Suhas: That is absolutely correct. (59:05)

Alexey: I have to run now so I want to thank you for joining us today and sharing your experience. We are going to raffle off a few copies of the book to people who signed up for this event. The lucky person who becomes member one hundred thousand in our community is also going to get a copy. (59:12)

Alexey: We are going to invite your coauthor for an interview soon to answer the remaining audience questions we missed today. Thank you everyone for joining this excellent discussion. (59:48)

Suhas: Thank you Alexey and I hope everyone has a great rest of the day. I am going to jump to another meeting that I am already late for. (1:00:17)