AI Adoption in Enterprise Beyond Writing Code | Ivan Bilan

The transcripts are edited for clarity, sometimes with AI. If you notice any incorrect information, let us know.

Career Journey in Data Science and NLP

Alexey: Hi everyone, welcome to our event. This event is brought to you by DataTalksClub which is a community of people who love data. Typically I have a slide here but I thought I would just skip it. Please like the video and subscribe. We have an amazing Slack community and the link is in the description. (0:00)

Alexey: The important thing is if you have any questions today to our guest Ivan there is a pinned link in the live chat. Click on that link to ask your questions so we will be covering them during the interview. It is my big pleasure to invite Ivan again. Ivan was here already two or three years ago but it was a very lovely conversation. One day I was coming home from the gym. I was thinking NLP has changed so much over the last few years. (0:11)

Alexey: Probably it would be nice to get somebody on our podcast where we can talk about how NLP changed. The first person I thought about when thinking about this was you Ivan because you had this NLP what was the name? (0:54)

Ivan: It is computational linguistics which is what I studied which is a NLP thing. (1:13)

Alexey: Oh yeah I have the repo here. NLP pandect. It is basically a list of all things NLP. Usually people just call these projects awesome something awesomeness but you decided to call it differently. Did you decide to call it differently? (1:18)

Ivan: Can you say it again what is the name? It is NLP Pandect. It is on my GitHub if you search for my name. (1:34)

Alexey: What does it mean? Pandect. (1:41)

Ivan: It is old Greek for encyclopedia. (1:43)

Alexey: Did you have Greek letters all over the place? (1:49)

Ivan: Yes. I just wanted to make something different to awesome lists because there are so many awesome lists. (1:53)

Alexey: With time you got promoted and started working as an engineering manager. This is what the podcast we had was about getting from individual contributor to an engineering manager. I reached out to you thinking whether you can talk about NLP but then I realized there is something more that we can talk about. Now everyone is obsessed myself included with all this generative AI and what it can do to help us. I keep hearing from people that expectations from the team members and also from managers because of that are growing. (2:01)

Alexey: I thought Ivan you would be the best speaker to talk about that. Here we are talking about this. It is very nice to have you back. (2:39)

Ivan: Happy to be here. Thanks for inviting me. There is a lot to talk about is there not? We planned this maybe a month or two ago and it feels a light year since then. So many things have changed. (2:45)

Alexey: I need to formally introduce you. Hi everyone. Ivan is a senior engineering manager at Personio leading multiple teams in the identity and access management domain. Every time I see this IAM three letters I have PTSD from my time working in a corporation and dealing with all this IAM stuff in AWS. It is a very important thing. (3:04)

Alexey: Previously Ivan served as a data science and engineering manager at TrustYou where he led NLP and infrastructure groups to optimize massive ETL and ML pipelines. With a masters in computational linguistics from LMU Munich and a background in CDTM I have no idea what that is. Ivan bridges the gap between deep technical NLP research and senior leadership. Can you tell us what CDTM is? (3:36)

Ivan: That is actually part of a technical university in Munich. It is a technical leadership and management degree that they do additionally. I studied computational linguistics at the same time doing an MBA but more engineering specific. (4:02)

Alexey: Are you still based in Munich? (4:20)

Ivan: Yes I am still in Munich. (4:27)

Alexey: I will be in Munich in a couple of weeks. It would be nice. (4:32)

Ivan: Nice. Thanks for the intro. (4:35)

Ivan: I have worked a lot in AI before. My master thesis was on the transformer model. I was doing some tweaks to it way back when it first released. Then I moved a bit to a different area. You cannot escape AI even in IAM you still have MCPs now which is authentication for AI agents. (4:39)

Ivan: Obviously everyone is now using AI for so many things. We can talk a bit about how industry adoption of AI is going on now. I have been talking to a lot of people working in different organizations of different sizes trying to understand what is happening. If you look at the news every week something new comes up from security breaches to crazy IPOs and whatnot. (5:04)

Alexey: We will definitely talk about all that but I am really curious to know more about you. I already know a few things about you but maybe for our guests you can tell us about your career journey so far. We briefly touched it when I was talking about your biography so far but maybe you can give us some more proper introduction of what you have done and how you ended up doing what you do right now. (5:37)

Ivan: As I mentioned I studied computational linguistics which is basically data science and NLP based. Then I was working for a while as a data engineer building ETL pipelines. People will remember Hadoop Spark and stuff like that. That was a lot of fun back then. Then I moved more into doing data science specifically. (6:07)

Alexey: You know how fun it was to do and build those ETL pipelines. (6:31)

Ivan: I was doing sentiment analysis summarization and a few other things in NLP. I worked on a multilingual system where we supported twenty languages back then. That was quite interesting working a lot with Asian languages as well. NLP for Asian and Arabic languages is super complex. Then I moved on more into technical leadership started working with different kinds of teams. (6:37)

Ivan: A great opportunity came up to move into something more security focused. I was always interested in that and for five years I have been working in identity and access management space doing access rights and login. Now it all comes back to AI again. AI agents need access right checks and proper MCP authentication so on and so forth. No matter where you work now there is no escaping AI I would guess. (7:07)

Industry Adoption of Generative AI and Agents

Alexey: I went to a friend recently with my son. My son is very inventive. He has a pair of pants and there was a hole on this and he fixed it with scotch tape. He just put scotch tape on the pants. I saw you are growing up to be a developer. (7:37)

Alexey: We developers use these kind of patches. He was saying he does not want to be a developer. We were at my friends and my friend said that actually developer is an endangered species right now. With all the AI stuff my son said he wants to work as a mechanic with gears and stuff. It seems to be a safer job these days. (8:07)

Alexey: Do you think developers are going to be needed in five years? (8:42)

Ivan: That is an interesting one. End of last year that was a take that we are going to chill and AI will do the work for us. It is completely different if you talk to people working in the industry right now we actually have more work than we had before. We now have to manage AI agents and put everything they produce together. What is also happening in the industry is that now that you have these AI helpers you work a lot in parallel things. (8:47)

Ivan: Maybe before that you had one work stream focused flow to work on that. Now you try to focus on one thing and in the background ten things are happening. Your AI agents are doing something and then you actually need to go back and be a bit of a manager. Even as a software engineer you need to do a bit of management now but you are managing AI agents basically instead of people. (9:20)

Alexey: For you this is something you are used to I guess when you were an engineering manager you had to switch contexts all the time. (9:45)

Ivan: For engineering managers that is what we do but I think for engineers that is a new kind of change. It also takes time for engineers to get into that flow. We need to give everyone enough time to experiment because there is just so much to learn even for engineers. We went from basic prompt engineering to now talking about context engineering. We are talking about teaching AI agents to be effective and to use less tokens. (9:50)

Ivan: There are just so many things everyone needs to figure out right now and there is a lot to learn. As I mentioned things are changing almost all the time. (10:35)

Alexey: You mentioned Hadoop. Probably you remember old days where everyone was doing things differently. Over time we as an industry converged to using Spark and nobody was writing my previous jobs anymore. Now people do not use Spark anymore so it is considered legacy. What I am trying to say is right now things are about how we teach our agents to be more effective and use less context or fewer tokens. (10:43)

Alexey: This is the early days of Hadoop where people were thinking how do we do map reduce in a way that we can actually manage this complexity. Over time things became easier and all we needed to do was to write a spark job and spark would create all these map reduce things itself we would not need to think about this. Do you think this is where we are also going with AI? (11:12)

Buy vs Build Dilemma for AI Infrastructure

Ivan: Probably. Bigger companies that have a bit more budget have a headway because they have a bit more time. There is a big decision for companies now whether to pay quite a bit for readymade models or start doing something local. Some smaller companies would need a lot of time to ramp up. They need to set something up like some local models and some sort of orchestration around it which takes a lot of time. (11:45)

Ivan: Bigger companies can go in and pay for a subscription which is expensive at the start but everyone can jump right in. You are using cloud codex or cursor which is much easier than waiting for your infrastructure team to set something up with deepseek or whatever. A lot of these bigger companies are really pushing now and I think a lot of standards will come out of that. You can kind of see that already as big companies are starting to share more what they are doing. We are looking at them and a good example is Amazon. (12:15)

Ivan: They had their problems in December where they tried something where AI does everything and then it backfired to some extent. Now they changed the policy where AI code needs to be reviewed by a senior engineer. These big companies try out these massive experiments and they are in the forefront of giving us the guidelines to some extent. It is also on all of us to keep experimenting and things will be changing probably as well. (12:55)

Alexey: Do you think software engineering is a safe career choice for kids these days? (13:28)

Ivan: I am pretty sure. There is just so much work. I have some friends who are freelancers and their business is booming because everyone coded something and it does not scale. It does not work and they just have so much work. My friend who is a freelancer has never had that much work in their life. (13:34)

Alexey: For us it is good. The code that AI generates is sloppy does not scale and there are many problems with that when engineers use it. I think we as engineers tend to produce code that is scalable. At least I want to think this way that the code we produce through agents is probably more scalable. Do you know how true that is? (13:58)

Ivan: I think there is a lot. We kind of went from this idea of AI slop to now figuring out how to actually through context engineering make AI agents produce really good output. If you give all the right context and if you have everything lined up for AI agents or the model that you use the quality is actually pretty good. You need to invest a lot of time to build that context and maintain it. You need to do some pre work. (14:26)

Ivan: Make sure your code documentation is actually next in the repository not in some Google doc. There is a lot of work to be done and I think there is a shift now where we are moving away from this idea that AI does sloppy work. AI is actually doing pretty good work now if you invest the time into making sure it can actually do the work right. That costs time and the output is pretty good after that. You still need to manage all of that and as I mentioned many things in parallel are running so you need to manage that as well. (15:02)

AI Capability Limits in Fixing Tech Debt

Ivan: You need to constantly maintain that. I feel like AI cannot do everything yet either. AI is pretty good at fixing bugs doing migrations and fixing tech debt but not building something from scratch. You either go and spend hundreds of agents to fix all of that and then manage that or you actually just do it yourself probably within the same amount of time. (15:46)

Alexey: There are cases where I just need to stop the agent and go check it myself. I would just find a few files delete them all and say now reimplement everything is deleted. Do not even try to check the Git history. Now we need to reimplement it and this is how we should do this. Then I explain almost line by line and after that it works. (16:17)

Alexey: These cases are more rare now these days where I need to nuke the code base and say now you need to rewrite this part from scratch. (16:42)

Ivan: It also depends on the organization setup. If you are working in a larger organization where there are just many microservices then AI can get that context faster than you and help you contribute outside of your domain. If you are working just on one repository or you are building your own project then it is pretty easy to handle as well. (16:53)

Alexey: What do you do as a senior manager? (17:13)

Ivan: Good question. (17:19)

Alexey: You ask a manager what they do. Now you manage managers. Do you manage managers? (17:24)

Ivan: Not right now. It depends and it changed a bit. I mostly work with more senior people on staff engineers and technical leads. I do some stuff hands on as well mainly taking over some data analytics for my team. We want to understand the data we get as well. (17:30)

Ivan: I basically work on what we are doing on a company level as well to some extent. I figure out what we are building and how my team can support. There is hands on work as well for building new features. (17:49)

Alexey: The reason I asked this question is not to put a manager in an uncomfortable position and try to make them explain what they do. I also was a manager and at the end of the day you wonder what you did today. You are constantly context switching. The reason I had this question is I wanted to lead with this question to another question which is how much hands on work you do now. Everyone is talking about December last year as that time when agents became useful for me. (18:08)

Alexey: I started using them earlier so for me December was just a usual month. Let us take December half a year ago. Now compared to before is your work different. Do you do more hands on stuff? (18:48)

Ivan: It depends. Smaller things yes. That is kind of what I hear from the industry as well now. You saw a bug two years ago and everyone tells you for two years there is no capacity to fix it so you just go ahead and do it yourselves now. You can spin up AI agents and fix that and release it so I have been doing a lot of that as well. (19:06)

Developer Workloads and AI Code Contributions

Ivan: With that there is an interesting thing developing regarding whether everyone should be contributing code now. Any role could contribute if it is possible and if the company setup allows for that. An interesting thing that happened is now when people actually try that they find out that writing code is just one of the ten things you need to actually do to push something to production. An AI agent can code something for you but then tests fail CI CD fails rollout breaks or something breaks. That is where actually most of the complexity is. (19:32)

Ivan: I have seen some companies realize that and then you have two options. You either teach everyone all of the concepts not just coding but how to be able to fix CI CD and where to get that context. Or you just say like what Amazon did that a senior engineer needs to review and nothing goes to production if no engineer actually reviews it. (20:19)

Alexey: Maybe there is a middle ground. Everyone can deploy their code but to some sort of sandboxed environment. There you do not need any approval from senior engineers. If it is production then of course you need a real experienced person to take a look at this and approve or reject. (20:55)

Ivan: That is true. That also generates more work for engineers. We were talking about how the workload grows and the workload also grows because now you are getting MRs and you have to help. You have to not just review but sometimes invest the time to help so that next time the person contributes they can do it on their own. That is the upfront investment that if you want to allow that in your organization you need to set off some time for that upfront investment. (21:19)

Alexey: Right now I have two questions. The first question is how do you actually do this. Before that there is this guy who created open claw from Vienna. Do you know his name? (21:54)

Ivan: I forgot the name but I know who you are talking about. (22:05)

Alexey: Every time I open Twitter I see this person and he is kind of humble bragging saying look I spent one million on tokens today or something like this. All these merge requests. His scale is unprecedented and other people do not do this but the thing is he has a process where there are agents that are working on these PRs. The agents are closing the PRs the agents are reviewing the PRs the agents are rejecting PRs and these PRs are created by other agents. It is a very interesting situation that at least for his project he needs to work on. (22:12)

Alexey: He has a setup for that. Coming back to where we started and why I thought about this you mentioned that we need to educate people and have the proper setup for that. We create more work for engineers but how do we manage this work. This person found a way to manage the work with open claw and all these codex agents that he is using. In your case and maybe people you talk to what is the typical setup to make it possible for developers to actually bear with this load? (22:47)

Ivan: Good question. It is important to make the LLMs or AI agents that you use actually useful. You need to go in and do context engineering to give them access to everything they need and have clear instructions. Otherwise you are going to be wasting so much time on fixing stuff after the AI agent so you need to spend time up front. Then you can go a level higher. (23:37)

Ivan: I see a lot of interesting setups that are open sourced where you have different types of agents. There is a project on GitHub called three man team. One person is a developer agent the other one is a reviewer agent and the third one is a product agent or architect. They all have separate roles. One creates the code the other one solely focuses on reviewing it and making sure it fits the architectural design and the third one cross checks it with product requirements. (24:10)

Experimentation with Open Source AI Agent Architectures

Ivan: That is just one example. That is where we are now. We need to keep understanding how to make AI more efficient through these more elaborate setups while making sure we do not overdo it. We must ensure we do not have a huge memory problem in terms of the AI forgetting context management. That is quite a lot and then on a level I would say really what is important is you need the time to experiment. (24:49)

Ivan: If you are adopting AI I do not expect it to be immediately next week everyone is using it. Everyone needs the time to experiment and play around with it and try different tools and different setups. Then eventually converge on something. There needs to be that support. (25:25)

Alexey: Why do you think it does not work this way that you just hand everyone a cloud code subscription and they start using it in a week. What is preventing them from using it. They can use it. Can they use it? (25:45)

Ivan: Will the output be useful first of all. What will probably also happen is that everyone will start building their own plugins and things where you could actually unify that. You can make sure everyone is converging on similar tools and similar plugins. Because then everyone wastes time building their own thing and then they need to maintain it. There is also varying quality where someone did it better and the other one maybe has lower quality. (26:04)

Ivan: Eventually if you are using stuff like Claude code or whatever the setup you need to work on unifying that. Maybe create a general strategy for the company. (26:44)

Alexey: A more pragmatic or wise solution in this case would be to just get a few senior engineers and give them a task. You use this for a month and figure out what kind of infrastructure we need in order to scale it to the entire company. Then they go deeper into this and come up with some sort of best practices. It is unified. (26:51)

Ivan: I come from what I was doing before. I ran an MLOps team. (27:21)

Alexey: This is what we were always doing. We want to introduce a new technology so now we need to actually understand how people would use this technology. What kind of things they would need on top of that not just locking yourself in a room for a month and then going out and saying this is how you should use this. You constantly talk to people and get their feedback and improve. Probably this could be a way when it comes to using organization. (27:28)

Ivan: That is also true. There are just so many ways and I think so many companies do it in different ways. That is what I meant about these bigger organizations experimenting. You either wait for them to share how they did it and what actually worked or you go in and experiment yourself and see what happens. (27:56)

Alexey: What is the role of a manager or a senior manager in this AI adoption? (28:22)

Ivan: I think it is basically ensuring that it is actually useful. Making sure what we mentioned before there is a unified approach to it so that we are not in a state where everyone builds their own thing on top of whatever LLM you are using. The whole team does stuff differently and it is more about unifying things and making sure the team is aligned. Ideally aligned with this company direction and how that is set up and using what the company is using as well. For managers it is a big question now whether it is actually useful. (28:30)

Ivan: How do you track that. How do you check if we are actually bringing more value to customers and if customers are even noticing anything changed. For internal developer experience AI is good for fixing small things tech debt and migrations so you see where you can apply that. I am pretty sure there is no team in the world that does not have a tech debt backlog. You see how you can apply it there and if you can start fixing some things there that you just never really had time for. (29:11)

Ivan: Where is the usefulness of AI you need to check that and apply it in the right place. Then converge the team to use the same tools. (29:51)

Alexey: How do we measure if it is useful or not. There is a story probably also heard from Uber where they burnt through their entire year budget and now they ask themselves if it was actually useful. My understanding from what I read online is that they could not give a satisfying yes. They kind of spent all this money but what did they do? (29:58)

Measuring ROI and Business Value of AI Integration

Ivan: These big companies also went the way of having no limit. Even worse there is a leaderboard to see who spends more tokens. People just run LLMs in a loop doing some random stuff just to generate and burn more tokens so that you look better on the leaderboard. (30:06)

Alexey: That is horrible. Is it horrible? (30:54)

Ivan: Yes and I think we are finally going away from that. That was an expensive experiment for Uber and the others but I think everyone realizes now that that was not a great idea. Basically how do you measure that which is hard. Ideally there is nothing you need to change you should already have measurements of the quality of your product from customers like NPS or whatever you are tracking. Also the satisfaction of developer experience from engineers we all should be tracking that ideally already. (30:59)

Ivan: If you already have all of that in place then you actually should see the impact in those metrics. The question is how good those metrics are in your company and if they are measuring the right things. I would say customer satisfaction obviously and then the satisfaction of developer experience inside the company. (31:44)

Alexey: Do we have actually because this putting agents on such scale is a relatively new thing. We did not have everyone talking about using teams of agents a year ago it started recently. Do we have actually enough data to say that we did this experiment and it led to the improvement of NPS. NPS is net promoter score. Do we have enough data to actually say that or maybe these experiments do not need to be long running. (32:02)

Alexey: Nothing stops us from shipping the feature in a week and customers are happy and we continue shipping. Or maybe we should stop and think about things. What do you think about this? (32:41)

Ivan: Everyone is trying to figure it out now and some companies are still in the mode where they throw money at it and see what happens. For smaller ones that is not an option so they need to be more careful with that. They need to make sure either they set up some local setup that they manage where tokens are cheaper for them or they are really checking what the progress and impact is and adjusting accordingly. I do not think there is a clear answer yet because we are still in that hyper adoption mode. (32:53)

Alexey: You mentioned token consumption and leaderboards and this is the worst possible metric for that. It is like measuring the output of a developer in lines of code we all know that it does not work. Do you know any other metrics that works okay at the end. The only thing that matters is in the case of business is revenue and whether the revenue is growing. How exactly are you affecting this revenue through some other proxy metrics like NPS or team satisfaction. (33:42)

Alexey: Are there any other metrics that are closer to maybe this token consumption. In order to measure how exactly you affect the revenue it is a long journey and we might not have enough data to do this. There are some other metrics that could be useful like number of pull requests or number of merged pull requests. Which kind of developer specific metrics have you seen work better in practice than token consumption? (34:24)

Ivan: Good question. We are going back to the discussion of these developer metrics like DORA and similar metrics. You mentioned pull requests and so on and I am also not very convinced those are so helpful. I think it really depends on what you are doing and what you are focusing on. If you say you are going to throw AI into fixing tech debt then a good measure would be how long your CI CD pipeline runs and how long does it take from creating an MR to actually releasing it. (34:48)

Tracking AI Impact Using DORA Metrics

Ivan: Are you actually improving that time and is it faster or better. Is there less code complexity now to deal with because AI fixed all the tech debt hopefully. (35:10)

Alexey: Are there metrics to measure code complexity? (35:39)

Ivan: I think there are quite a lot but I have not seen them used really in the industry. (35:44)

Alexey: When I was a Java developer we had Sonar and Sonar would output some cryptic metrics which I had no idea how they actually work. When it goes down it is a good job. I guess this is one of those. Is it one of those? (35:50)

Ivan: Yeah. There is this AST thing in Java where it measures complexity of code but I have only seen it in research. I did some research on that as well but I have never really seen it used in the industry. In the industry it is more like going back to the DORA metrics. It was a few years ago a research done by ThoughtWorks where they did research on what are the best metrics to measure output of code. (36:03)

Ivan: I think most of those metrics are what you mentioned like MR throughput so how many MRs you get per week. Then measuring from the time you start and create the MR how long does it take until it lands with a customer. It is a cycle metric basically it just tracks how long does it take for everything like tests to run and the team to review the MR. It tracks how long it takes in CI CD delivery canary releases and whatnot. You measure from start to finish where worst case is a week or two and best case is an hour. (36:43)

Ivan: That is one of the metrics that we should be optimizing for because nobody wants to wait. (37:25)

Alexey: The regressions we had like number of bugs that are piled as a whole app that is also a part of it. (37:31)

Ivan: I think so. I am checking it right now but it is a very long document. The idea is you want to have software delivery performance metrics to show how good your delivery pipeline is. (37:36)

Alexey: Lead time how long it takes to production. (37:59)

Ivan: Then you mentioned this failure rate that is there as well so how often or how many bugs happen after that. How much time do you need to spend on fixing stuff afterwards and that is an interesting one for AI agents and uptime as well. For AI agents you deliver something and then checking what the failure rate of code generated by engineers versus failure rate of code generated by AI agents is a good metric to track. I have not really thought about that but as we were talking about that it is a good one. (38:06)

Alexey: Now with a lot of AI code being generated we do not know if it is just correlation or causation. With all these CI tools many major services like GitHub experience a lot of outages. If you go to their status pages you do not see this 99.999 percent deal anymore. It is in the 90s and in some cases it goes to the 80s I saw that. For GitHub they are in a very tricky situation right now because all these agents publish code on GitHub where all these CI CD jobs run. (38:44)

Alexey: They store all these artifacts on GitHub so it is very tricky for them. There are other providers like there was a story that a rsync maintainer discovered Claude code recently and now with the newest release this thing stopped working. (39:29)

Impact of AI Code Generation on CI/CD System Reliability

Ivan: Some companies just want to go fast and failure rate goes up but they are still delivering new things. I do not know. I have seen that about GitHub. The same thing happened with Amazon where there were so many outages and then now it is kind of under control. Maybe GitHub enforces more code checks and that helps so we will see. (39:51)

Alexey: Essentially what I heard from you and you repeated multiple times is the big companies pave the way. They experiment and they are really on the frontier of all these things. They have the money and they have a lot of people to experiment or sometimes they are forced to be a part of the experiment like GitHub. What we can do as people who do not work in these big companies is we can learn from them because usually this information is public. Is it usually public? (40:22)

Ivan: Exactly. I think that is a good source. Thankfully they put out a lot of postmortems as well and they put out some blogs as well. It is interesting to see how that develops. (40:58)

Alexey: Let us say you join a new team and your task as a senior engineering manager is to set up the framework and processes in order for people to use AI as effectively as possible. How would you approach this? Let us say the company and the team does not use AI yet. The individual developers are trying to use this but there is no standard way of doing this. (41:17)

Ivan: I think the first thing I would do and this is probably controversial is to say you have two weeks to do anything with AI. There is no expectation to deliver anything. The expectation is that you experiment learn tools and then at the end of that two weeks we run a knowledge share. There should be no expectation that you actually build something because it is just so complex. (41:43)

Ivan: Give the team the space to try out things without fearing that they need to build something in two weeks and have no clue how. They at least start and they have the free time. What is important is continuous knowledge sharing. Some engineers will do it in one way others in the other and this is how it was at the beginning we all learned things. We all learn things differently. (42:11)

Ivan: We follow different people online we read different blogs and we try different things. Talking to my engineers they did stuff completely differently. They use completely different tools for context management. I use a different tool just because I found it myself or someone recommended it to me. You need to work on converging so you need to work on knowledge sharing and ideally you agree on one way of using AI if possible with some variation. (42:42)

Best Practices for Team AI Tool Adoption

Alexey: How do you do these knowledge sharing sessions. Is it like you all get together in a room and everyone presents a demo. Then everyone knows what exactly you did in these two weeks. (43:00)

Ivan: Show what you learned and show the tools. Then have a conversation about what you liked and what you did not like and what could have been better like obstacles. Also collect feedback like people will say they could have done the same thing at the same time themselves without using AI. (43:24)

Alexey: There are expectations for people to have a deliverable at this point but the expectation is they just try it and they share what they did. (43:36)

Ivan: I would propose that. I think that would work really well for the first two weeks and then maybe after that you can actually say now we have better understanding of it let us do a hackathon and actually build something. That is a follow up on that. (43:52)

Alexey: So a hackathon is for a couple of days. (44:05)

Ivan: Yes. (44:10)

Alexey: Everyone participates in a hackathon and builds something and then what. (44:15)

Ivan: It depends on the organization. Ideally you build something that is part of the product and a new feature. Then you actually test how close you are with the tools that you have to pushing something that is not just a prototype but something that can be delivered to customers. That is a good measurement of whether you are using the right tools. Maybe you are using some LLM that is not good. (44:21)

Ivan: Maybe you need to use a different one or you use different ones already as a part of the experiment and then you measure the hackathon project output. Is this just something that will never go to production. Then maybe you need to go back and check your tools and check what happened. Or if it is something that is working you clean it up and release it to production. That is a good indicator that the tooling is good and the team figured out how to do it together. (44:45)

Alexey: You will probably need a few iterations of that. (45:24)

Ivan: Definitely. (45:29)

Alexey: After you iterate what is the good state. What is the state we want to be eventually in. (45:29)

Ivan: The state we want to be in is basically it becomes part of regular workflow. It becomes a regular thing for the team to use the tools. The ideal outcome is that it becomes invisible whereas you still produce code and it is still good quality code. You might be able to produce a bit more depending on what you are using but ideally the quality should not go down. That is what I mean by invisible as the quality should stay so you need to invest in that. (45:35)

Ivan: You are just using AI then and the tradeoff of that is the workload of the whole team will then increase because there is more stuff to review and manage. Some work needs to be done then in terms of team processes. Do we change the team processes and how much throughput can we actually handle as a team. If someone does ten PRs per week and that is just not sustainable for the team then we need to scale down or scale up. (46:14)

Alexey: The goal is the tool is invisible and the code we produce is still good quality. Maybe we will do a little bit more but with this invisibility there is the cost. Then as a company you pay more and sometimes it is a lot more considering what happened now to GitHub Copilot. How do you then decide whether it is actually a thing worth keeping? (46:47)

Ivan: I think that is for every company to figure out their budget for this and probably set some limits as well. A good strategy would be you pay for a subscription but at the same time your AI engineering team works on something local fine tuned for your organization and less expensive. I think that is a good approach. We have seen in the last half a year the costs just go up and the features go down. I think they removed the coding feature from the Mac subscription so the features go down and the cost goes up and we do not know how far that cost will go because it is increasingly more expensive to run those things. (47:26)

Managing Vendor Lock-In Risks with AI Providers

Alexey: Another thing I wanted to ask you is the dependency but I think you answered that because as we start using it and incorporating it in our flow we become dependent on the provider. This happened to me personally I got excited about Claude code so much I started using it a lot and then at some point they had some problems. It was a few months ago where you would hit the weekly limit in one session or something. (48:20)

Alexey: I was wondering what I do now as my workflow is so dependent on this tool right now. I started thinking about Codex and other providers and how I make my work independent so I do not depend on the tool. This is just me as a solo developer but as a team you have a lot more risks. I am flexible I can just ditch Claude and start using Codex which of course will require some readjustment for me but I am more flexible as a single person. When you talk about a team of people or an entire organization you cannot just tell everyone today we use Codex and then in one week say actually we go back to Claude. (48:47)

Alexey: Is the answer having this AI engineering team working on the local solution? (49:37)

Ivan: For me I had the same problem. What I am doing myself and what others I see do is that for day to day stuff you use open source models. There are so many open source models and many tools you can use for that. When you converge on something and you want to actually build something production ready then you pay and get your subscription and use it. I do not know if that is the answer for big organizations but I would assume so if you want to save on costs. (49:43)

Ivan: You need to in parallel start building something yourself or something that is just cheaper. You could even use AWS Bedrock and use open source models there. The orchestration layer is there so it is easier to do something yourself and it is much cheaper. We do not know how much more expensive this stuff will get but I think it is like with any tool and not just AI. We have the same problems with other tools regarding how you manage your dependency. (50:20)

Ivan: Basically what you can build in an agnostic way you build in an agnostic way. Let us say you are building the Claude markdown file. Ideally it would be reusable in another LLM. You build it in a way that is not specific to Claude or whatever you are using. (50:52)

Importance of Hiring Junior Software Engineers

Alexey: In my case the instruction I have in Claude markdown is go agents.md. That is the only line in all my Claude markdown files. Now another question for you about juniors. We know that getting hired as a junior is much more difficult compared to a few years ago. AI is one of the reasons for that. (51:27)

Alexey: Now you can get your senior engineer a Claude code subscription and the senior engineer would produce more effectively. There is no reason for us to hire juniors. Eventually these engineers will leave their software engineering job and focus on carpentry or become managers. Eventually we need fresh blood so the companies probably also realize this but I do not see a big uptick in junior positions still. Why do you think we do not hire juniors anymore and what can they do to get hired? (51:37)

Ivan: That is a big problem I think. It always reminds me of a few people who maintain Prolog systems and they get brought out from retirement just to do that for a lot of cash. I hope that does not happen for us. It is important to onboard juniors into software engineering. There is also an interesting thing happening right now where you jokingly see people saying it is cheaper to hire a junior engineer than run Claude or Codex because the cost is higher than what you would pay an actual human. (52:25)

Ivan: We do not know how far the cost will go up but there might be a tipping point where you just say this is too expensive let us just hire people again. It is really important and I think every company needs to think about that and start hiring juniors. One thing I can say about my current company Personio is we started a drive to hire junior engineers. We have open positions now which is great and I think other companies need to think about that as well. That knowledge will go away and I think people still have this thought that AI is there and it can do the work but AI does not have the context. (53:09)

Ivan: You need someone to provide the context and you need someone to fix the problems and orchestrate AI. If in twenty years we all retire who is going to do the job? (53:57)

Alexey: Another thing I was thinking about is we senior engineers learned coding when we did not have AI assistants. Even if the circumstances force me I will be able to go and modify the code and update the code because I still have the skills. They are rusty now compared to before because I do not do this anymore I just give instructions but I still have some muscle memory. Juniors never did this at the scale as the older generation. What are the risks of hiring juniors with this kind of setup right now and should we think about this now or just hire juniors anyways and see what happens? (54:08)

Ivan: I think we should hire them. People do not use current LLM models enough to learn. I love doing that and instead of just coding something I ask a prompt to explain everything in detail and what are the steps. I learn a lot through that. Ten years ago I would go to Stack Overflow post and be called an idiot first and then someone would actually answer it. (55:06)

Alexey: It is part of the experience. You have to live through this. (55:34)

Ivan: Now you can just go and ask an LLM for it. I think the other problem is to what extent will LLMs keep giving you the right answers. We have this idea of model collapse going around where at one point there will not be enough training data for LLMs. To what extent will they be helpful here I do not know. Just because of that it is easier to start as a junior right because you no longer have to come up to some senior person and say I need help and sit down next to them and bug them. (55:39)

Ivan: Now you can just open Codex and start asking a hundred questions there. It is easier to onboard yourself with LLMs presence. (56:20)

Accelerated Junior Developer Onboarding with AI Assistants

Alexey: This is a comment that I just see right now that there is a theory the juniors catch up faster with AI. This is exactly your point. It also gives more breathing room for senior engineers. Senior engineers can actually focus on what they do and not be backed by people constantly bugging them. (56:28)

Ivan: The bottom line is I think we should keep hiring juniors but let us see what happens. It is going so fast we do not know how it will develop. (56:51)

Alexey: Do you have time for one more question? (57:01)

Ivan: Sure. Go ahead. (57:08)

Alexey: There is a question about people who want to start a career in AI engineering as a software engineer. Maybe you have some recommendations for them. As a manager you see people making these transitions. What would you recommend to them right now in 2026? (57:08)

Ivan: It depends. I think we still have quite a broad spectrum of AI roles. We still have regular MLOps where someone needs to orchestrate everything. If your company is doing RAG like retrieval then someone needs to build that. That is basically like we had ETL pipelines but now it is RAG. (57:28)

Ivan: You can go that way and learn the tools there. The good thing is there are a lot of open source tools like LlamaIndex Langchain Haystack and many others. You can just go and build a toy project yourself. Of course if you have a GPU that is great or you can rent something online. I feel like there is another role forming in the industry where you are focusing on context building. (57:55)

Ivan: You are actually the one writing very efficient instructions and building an infrastructure around reengineering prompts and context how it is given to the models. That is another way you can go. I am pretty sure that will stay for a while as well and you can learn the best practices there. You can experiment locally as there are just so many tools right now. I am really happy to see open source is still alive and well in AI. (58:26)

Alexey: It is catching up. There are amazing tools like Anything LLM on GitHub. You can plug in any LLM model there and play around with all the steps you need. (58:58)

Alexey: This context building and context engineering you mentioned so many times. It feels like I need to find somebody who would talk about this thing for an episode. It is pretty new I think. I heard the term of context engineering maybe a month ago so it is developing now. (59:09)

Ivan: That is this idea I have heard. We went from assuming AI is just going to keep generating slop to realizing that is not a problem with the model but the problem with orchestration around the model. You need to spend more time improving everything around the model like better prompts and better context. That is where context engineering comes into play. If you have everything set up well then the power of your LLM is much higher and the quality is better. (59:32)

Specification-Driven Development and Context Engineering

Alexey: There is a comment about specification driven development and I think this could be exactly that. You give your agent the context on how to approach a problem. There is a product manager who is scoping or grooming the problem. There is a software engineer who is working on implementing this. There is a tester who is testing this and then again I am accepting or rejecting this thing. (1:00:12)

Alexey: At least this is the setup I arrived at and this is the setup I use and you mentioned this too. I think we are probably talking about the same thing and it is just different people calling it different names. Is it part of the whole thing? (1:00:35)

Ivan: It is the building blocks of proper context engineering that are still being figured out. I mentioned this before that we are kind of in this research state where we do not know yet what is working best. (1:00:46)

Alexey: To me it is also about giving your agents ways to understand your codebase faster. I do not have an answer for that but this is how I understand it. All this code agents.md files and things like that. You want to say instead of doing grep on the entire codebase if you need to make a change you go here. Maybe you have an index on top of your codebase and I think Cursor is doing that. (1:01:04)

Ivan: Exactly. That is really helpful and at the same time also dangerous in terms of costs so you need to be very careful. It needs to be very precise. Every single time you do a prompt the LLM will load that context and spend tons of tokens on processing it. (1:01:27)

Alexey: Thanks a lot. It is a pleasure to talk to you as always. Thanks for staying a bit longer with us. I really enjoyed this conversation and thanks everyone for joining us today and asking questions. It was really nice. (1:01:46)

Ivan: Same here. Thanks a lot for inviting me. (1:02:04)

Alexey: Thanks for being here. I will drop you a line and maybe we can meet in Munich. (1:02:10)

Ivan: Definitely. (1:02:15)

Alexey: Speak soon. Bye. (1:02:15)

Ivan: Nice. (1:02:15)