Decoding the Open Lakehouse | Sahil Walia

The transcripts are edited for clarity, sometimes with AI. If you notice any incorrect information, let us know.

Defining the Modern Data Platform

Alexey: Hi everyone. This is our PK event. We have an interview. If you want to find out about the events we have there is a link in the description. Check it out. You will see all the events we have in our pipeline. Like the video, subscribe to our YouTube channel. There is also a Slack community where you can hang out with other data people with us. The link is also in the description. If you have any questions there is a pinned LinkedIn live chat. (0:00)

Alexey: Click on that link to ask your questions and we will be covering these questions during the interview. Today we will talk about open lakehouse. We have a very special guest today Sahil. Sahil is a senior technical architect at Snowflake where he works with enterprise customers running Apache iceberg lake houses with a focus on data engineering and AI ready platforms. He is also a maintainer of Schema Change, an open source database migration project with nearly 500,000 downloads per month. (0:28)

Alexey: He also contributes to Apache OSI an open source project for centralizing semantic model exchange. For me this interview is especially interesting because I am not professionally a data engineer. I have done a lot of data engineering as a data scientist just because I needed it. In order to create a project and make sure a data science project runs I needed to do a lot of transformation. I needed to do a lot of work in Spark and Presto and similar things. (0:52)

Alexey: I am a bit out of touch with all these modern things. Sahil it is very nice to have you here we can talk about all these things that modern data engineers need to use. Welcome to our interview. (1:29)

Sahil: Thanks for having me. I am super excited. (1:41)

Alexey: Make it a bit louder I can hear you well. Now it is great. Before we start and go into our main topic to talk about open lakehouse I want to talk about some of these libraries that you maintain and contribute to. I want to learn more about your background. Can you tell us more about your career journey so far? (1:44)

Sahil: Sounds good. Absolutely. Hey everyone my name is Sahil. I am based out of the northeast US. It is right at six thirty in the morning. (2:06)

Sahil: I am very fresh right now and very excited. Good morning. Let us start with my background. I have been in the industry for over a decade now especially doing data engineering. My job essentially started just any other SQL heavy role very much analytics heavy. (2:14)

Sahil: I was doing a lot of reporting and those kind of things. Early on when I started I was building reports writing simple SQL queries and learning joins. I was very new and enjoying that. Then I moved on to other roles wherein for example one of my roles at Ernst Young was automating parts of the gap reporting. They have some codes that they have to comply with where correctness really mattered in terms of not only the data having to be correct but also performing well. (2:35)

Sahil: That is how I started building. Then I moved to a company called Slalom in the consulting space. I was architecting modern data lakes moving heavy legacy systems to newer systems at that time say Snowflake. I was building those important pipelines and data quality frameworks. My role over the years got more technical and I realized that I am more of a technical person who enjoys that. (3:07)

Sahil: That kept happening and then I noticed that a lot of companies are dealing with many engines that have many opinions. Each engine has its own opinions on how they want files to be structured how they want performance to be curated how they want pruning to happen and those kind of things. That basically led me to all of this. Anyway I am at Snowflake right now as you mentioned Alexey. I am a senior technical architect here. (3:39)

Sahil: I have been here since 2024. I directly work with customers who run Apache Iceberg lakehouses. I am usually in the room talking mostly about data engineering and adjacent fields. We discuss what will consume the data where the data will come from how the pipelines would run and most of the times performance. We figure out how we make things better cheaper and faster. (4:08)

Sahil: That is pretty much about it. As time permits I try to contribute to Schema Change a very new project Apache OSI. I started contributing there. It is a very new community that I started working at. (4:32)

Alexey: That is really nice. You mentioned that one of the things I noticed when you were describing your career was that when you were consulting you were helping clients build modern data lakes. I have heard this term modern data platform. It has been around for quite some time. What do we understand by the term modern data platform and how modern is it actually? (4:50)

Alexey: This term has been around for quite some time. Is it still modern or do we just not have a better name for that? (5:16)

Sahil: I think that is a very fair question and it is modern in the context of where you are living. I will give you an example. There was a time when X platform or Y platform was modern at that time but as technology evolved a lot of those platforms did not evolve with it. A lot of things keep evolving. (5:24)

Alexey: Is this modern data platform more about technologies or is it also about the approach and how we do this? What is not modern or what is the old data platform? I guess the term modern data platform appeared because there was an old way of doing this. Now we say there is a new way of doing this and we call it a modern data platform. (5:57)

Sahil: I would not specifically say a technology I would say the way of doing things. We started with data warehouses then we moved to lakehouses and now we are in the interoperable lakehouses world. Data warehouses were mostly schema on write and mostly transactional. Then we had the world of lakehouses wherein people were just dumping data into S3 buckets but there was no structure around it. There were no good practices around it. (6:21)

Sahil: When we evolved we arrived at the current state which I would say is interoperable open lakehouses. Interoperable means that you want your data to be decoupled from compute. Your storage is decoupled from your compute and you are also making it open. Data engineering is very much motivated by what is happening in open source. A lot of things that happen in data engineering are inspired by open source. (6:52)

Sahil: For example storage is Parquet which is open source. Compute starts with Spark which dictates a lot more things. Similarly table formats for example Iceberg are open formats. They tell you how you can move faster. Companies like Snowflake maintain so much rich metadata on top of these open platforms that make them even more modern. (7:33)

Alexey: What is the more modern part here and what exactly do we compare? Previously one option we had was dumping all the data into whatever object storage we have. We figure out how exactly to translate it and let analytics engineers model this so they can do analysis. Is this what the modern approach is or is it something else? (8:08)

Sahil: This is a very difficult question to give one definition for. Modern can mean several things here. For example does it deliver to your modern sinks? Previously a lot of the analytical work we were doing as data engineers was mostly for reporting. Now it has evolved that your files or any data you prepare has to be ready for ML pipelines. (8:43)

Sahil: When we say modern we want to make sure it is contemporary which means it meets today's needs. Second are we making sure the platform we use does not lock you in? Object storage is your IP and that is where your things live. You should be flexible so as things become more contemporary there will be a new technology tomorrow. You still own your data and you just port it there. (9:15)

Alexey: Let us talk about this lock in part. Let us say I store data in Oracle and I do not have this interoperability because my compute and storage are the same thing. Once I keep my data in Parquet and store my data in S3 or Google Cloud Storage I can run whatever can reach this Parquet to do the analytics for me. Many engines can actually do that. Are we aligned? (10:03)

Sahil: Yes. You decoupled the storage and the compute. Now you have all the leverage you need from an architectural point to move things to a different layer or do compaction on the files. We can talk a little bit more about that when we dive into Iceberg but essentially you now own your own world. In the old world if you wanted to do something architecturally with the data on the physical layer you could not do it because you could not control the packaged platform. (10:33)

Sahil: Now that is going away. A lot of companies using these platforms are pushing the open source community because we need something where we own the data. We can do things around it and become more independent. (11:14)

What Exactly is a Lakehouse?

Alexey: Now I have a system generating a lot of JSON events. I just dump them to S3 and run Presto on top of that. Do I already have a lakehouse or is there more to a lakehouse? If I take this JSON and convert it to Parquet is it already a lakehouse or not yet? What actually is a lakehouse and where does it start? (11:34)

Sahil: When you ask that question let us think through this. What you are describing is a file storage. Parquet is very rich in metadata. It has file statistics showing this is columnar storage and it has its own row groups to maintain data. Parquet is very good and you get all the things you need from your data file. (12:01)

Sahil: Now let us talk about an engine. Let us take Spark as an example. How would Spark know what file to read? How would Spark know what file to prune? We need a few additional things around here like a catalog. (12:45)

Sahil: I can come to the catalog in a bit but we also need some metadata about the data. Metadata by definition is data about data. When we come into this world of open lakehouses you need some data around these files. You have ten files but what are they about? Are those about orders customers or revenue? (13:04)

Sahil: Which one are the latest files? From a logical aspect you need metadata about it. Now let us come to the catalog. The simplest way of explaining a catalog is telling an engine where your latest state of that table exists. You started with a small company with one hundred customers but now you have a thousand customers. (13:34)

Sahil: You want to make sure when you are querying today you get the thousand customer list not the one hundred customer list. You should still have the ability to go back in time to see those one hundred customers too. That is the job of the catalog. To answer your question Parquet are just files but you need a little bit more metadata on top that gives you clarity or navigability within the data for the engines. (13:58)

Alexey: If I understand you correctly with Spark you can just point it to read from an S3 location. You give it an explicit path so there is no catalog involved. You just say there are some files here read them process them and save the files in another S3 location. If we want to query customers where join date is 2026 this will not work without a catalog. We need the mapping from these table names and partition names to the actual data. (14:23)

Sahil: That is some of what the catalog is doing. Let us take your previous example when just looking at S3 storage. You are doing the work because you know that S3 location but the engine does not. When you do a select star from a table you are not doing anything you are just focused on what you want to get the results out of. The very simple way of thinking about a catalog is it is the location where your latest file exists. (15:14)

Sahil: That is not the only thing it does. Alexey is querying that data because he owns the company. What if I am querying that data but I do not own the company? I should not be going to those data files so I need to have controlled access. Permission management is another thing it does. (15:53)

Sahil: Now let us say YouTube is querying that particular file and you want to query that from Apple Podcast platform as a different engine. That other engine should not have access unless you specifically grant it. You might just want to provide a timed access to go to a specific location for only sixty seconds. Those are the kind of things that a catalog will additionally do. Iceberg is rich metadata about the files. (16:17)

Sahil: When you talk about querying where dates are greater than or year is less than those are the kind of things Iceberg will handle. It will do all the magic behind the scenes because it maintains a tree structure for the metadata. It goes into the nested layers and only gives you the data you need efficiently. (16:50)

Comparing Hive Metastore and Apache Iceberg

Alexey: I remember using Hive catalog and Glue catalog. They have been around for quite some time before Iceberg even appeared. I know that with Hive and Glue you can have this mapping from table names to file names. Can we already do permission management with Hive and what does Iceberg add on top of these things? (17:20)

Sahil: I will start by saying I do not have a ton of experience with Hive but I think you are talking about Hive metastore. Hive metastore maintains the naming mapping. In Iceberg it is very different and very advanced. It can do that plus extra things. The way it started was to resolve a lot of things at Netflix. (17:53)

Sahil: Netflix was going through a lot of issues having petabyte scale data. They just could not reach the right file at the right time. With Hive if you want a certain file sitting in an S3 bucket with a thousand files it has to read all thousand files under that sub path. It was not rich in metadata. With Iceberg it maintains a metadata JSON file at the very top level. (18:22)

Sahil: It talks about the latest state of the data called a snapshot. Within that snapshot it has manifest lists containing what file got changed and how many rows were added. It maintains file min and max values and then goes to the file level statistics. It does not have to do HTTP calls to list S3 contents every time to see what is going on. It will just go to its own metadata first to know what the files have and then only go to that specific file. (19:11)

Alexey: That is clear. At the end Iceberg is something similar to this Hive metastore but we have more granularity. We know more about where the files are. (20:01)

Sahil: I will explain that but I would segregate it into two things. Iceberg by itself is a table format. When we say Iceberg it is not a catalog it is a table format. The Iceberg REST catalog is an open API that is more similar to Hive metastore for clarity. (20:18)

Alexey: From what I understood there is a REST API that you can query. Let us say I want to calculate revenue for customers that joined this year. If I have a lot of customers like Netflix I need to be able to include customers that just joined. If they are joining a lot I need that granularity to know to include these customers when executing the query. The metadata manifests contain this information and all of that I can get through this REST API. (20:42)

Sahil: Correct. (21:29)

Alexey: At the end is it just a bunch of Parquet files or is there something else? The catalog is one thing and then we have Parquet and that is Iceberg. Are these correct? (21:29)

Sahil: Yes. The data is Parquet files. Your metadata file is one JSON file. You have some Avro files which maintain the data about the manifests. The metadata has pointers to the Avro files. (21:43)

Sahil: Ninety percent would be your Parquet files and you do not maintain any of the other places. That is something your Iceberg engine manages and it automatically creates when you integrate. You just need to take care of how you manage your data engineering pipelines. (22:07)

Handling GDPR and Data Deletions in Iceberg

Alexey: I remember when I needed to create a lot of data pipelines and all our data was in Parquet files the biggest problem was GDPR. What if somebody wants to be removed from the data of the company? It meant that we would actually need to go through all the Parquet files and recreate them. Does Iceberg solve this problem or make it cheaper? (22:34)

Sahil: GDPR is very strict. As I understand you need to delete the files as well not only the customer data to be forgotten. Iceberg is very strong at saying this is a particular file I do not want to reference anymore. It can create a delete file. It is very smart and will not touch your data files. (22:52)

Sahil: It will create a separate small protobuf file and attach it to that actual file. Whenever the engines are reading they will combine the statistics. It knows the whole file but understands it needs to delete these specific records and provides the readers the accurate picture at that time. That is what it does at the logical layer. (23:38)

Alexey: Your data still is in the physical file. In case you need to actually delete the file you have to do a little bit of operations. You delete the file and create a new file without the deleted data. (24:05)

Alexey: This thing is called a tombstone. I remember reading about tombstones in Designing Data Intensive Applications. The book describes this approach where you do not delete but create a tombstone so the data is kept there but cannot be accessed. At some point maybe once per month you collect all the tombstones and create a small file without any tombstones. I think it is called compaction and Iceberg can do this. Is that accurate? (24:20)

Sahil: Exactly. Iceberg does do that and it is very good at it. Let us look at any managed offering here from a data engineer perspective. You are a data engineer who wants to work with data and make sure pipelines run. Dealing with orphan files or managing storage accumulation is something you do not really enjoy. (25:00)

Sahil: These things are very trivial in nature and you do not learn anything from doing them every week. Data engineers do not enjoy it especially when things go wrong and you get pager alerts. Managed offerings have all of these things automated so they will take care of all of this for you. You just define a few things and that is the route I usually recommend if you do not want to take that burden. You still get the data that is yours but with a more relaxed mind. (25:39)

Sahil: It is just like having a lawn mower but paying someone to come and mow your lawn. That is the way I would think about it. (26:21)

Alexey: Apache Iceberg gives us this catalog and extra things on top of that like this compaction mechanism. It makes the lives of data engineers easier so they do not need to worry about all these things. In addition to Apache Iceberg it can give us what we call a lakehouse. If I have a setup where my files are in this format and I can query data using whatever agent that can access this data then I have a lakehouse. Are we aligned? (26:32)

Sahil: Correct. It makes everything open for you and it is one of the components. A lakehouse by definition is not just Apache Iceberg but Iceberg enables it. It has to have open storage like S3 or MinIO but I think you covered it all. (27:09)

Apache Iceberg vs. Delta Lake

Alexey: What is a Delta Lake? Is it an alternative to Iceberg or what is it exactly? (27:33)

Sahil: Delta Lake essentially came before Iceberg. Iceberg is relatively popular and the community has converged now that Iceberg is leading the way. Delta was started as open source and Delta lakes are based on Delta files. It was very well adopted but then Iceberg came in. Databricks is very much invested there which is why I am hesitant to talk too much about it. (27:39)

Sahil: I want to make sure I do not do injustice but the community has converged. Databricks does support Iceberg as well but there are a few challenges. Overall Delta Lake was good but Iceberg is generally considered more open. It was not driven by one particular vendor like Databricks or Snowflake. A lot of commits at Delta were motivated by Databricks but anyone can participate in Iceberg. (28:18)

Sahil: As a matter of fact Iceberg founders Daniel Weeks and Ryan Blue work for Databricks today. We have committers from Snowflake Apple Netflix and LinkedIn. Now it is truly a community project. It is truly open source and the Apache licensing is pretty strict. (29:18)

Alexey: As I understood with Apache Iceberg I can keep data in whatever object store and use whatever compatible engine. It can be Snowflake Databricks or BigQuery if they support it. I do not have this lock in. You mentioned interoperability and with Apache Iceberg I can get it. (29:42)

Sahil: That is right. (30:06)

Alexey: I actually wanted to talk about open lakehouse. Right now I have openlakehouse.io open in front of me. Are you somehow affiliated? What is open lakehouse and what is openlakehouse.io? (30:11)

Sahil: I think it is a company I am not familiar with to be very honest. Is that what you are looking at? (30:31)

Alexey: Yes it is sponsored by Databricks. I guess that is why you are not aware of them perhaps. What is an open lakehouse? (30:41)

Sahil: We did talk about lakehouses. Just a reminder for everyone a lakehouse is anything where you have structured and unstructured files that you can put in. The challenge was they were not very open and you wanted to do a lot more with your data. You wanted other applications to read it and write to it but you got stuck with one specific engine. An open lakehouse provides all those features where you can dump your data and organize it with rich metadata. (30:51)

Sahil: You have compaction time travel and ACID compliance. Lastly it can be plugged into any engine that supports the specification. The specification contains the rules on how data should be accessed written deleted and merged. One dilemma I want everyone to be aware of is that when we say open the focus is that data is open and can be accessed by other teams. (31:37)

Sahil: Iceberg tells you what things you can do in order to write or read. Write could mean updating a record merging a record or adding files. Both engines would say they can write to Iceberg but the definitions may be different. I would just suggest focusing on what your team does today or will do in the next few years and what kind of operations you need. Have a matrix to see what you want to achieve and whether a particular platform delivers that. (32:26)

Sahil: A lot of customers later realize a platform only supports features at a certain level. It does not mean they will not support it eventually but each engine has to come to that state of maturity. Open source leads it but ultimately engines have to implement it and think from their ecosystem perspective. (33:32)

Do You Need a Lakehouse for Small Data?

Alexey: I have a more practical question in mind. Let us say I want to build an open lakehouse. Let us take DataTalks.Club as an example. We have a few thousand events per day mostly people coming to the website and browsing. We also have a learning platform where people submit their homework. (34:01)

Alexey: I want to do some analytics on top of that. I want to understand what people are doing and how to convert visitors to community members. If people are doing the homework I want to know how to make them more successful. I want to have dashboards to understand where we are. Do I need a lakehouse for this? (34:34)

Alexey: Is this scale large enough to actually think about adopting Iceberg or do I need to be Netflix? (35:08)

Sahil: That is a very good question. At your scale if you are dealing with less than ten million records overall you should not even think about Apache Iceberg. (35:17)

Alexey: If my dataset fact tables contain less than ten million I should not even think about Apache Iceberg? (35:38)

Sahil: Iceberg makes you open and provides analytics but it has its own management layer too. When you are a smaller team doing things at a smaller level you do not want a lot of complexity unless you really need your data to be open. If you only have under one million records why would you go through all of this setting up just to get to the analytics? (35:47)

Alexey: It is not a one click setup that I am looking at. I need this REST catalog and there are some things on top of that. (36:32)

Sahil: To be very clear in Snowflake if you run a create table statement you just need to mention the catalog equals Snowflake. Snowflake will handle all of it even the storage for you. You do not have to connect to your S3 or any special catalog. It is very simple but the question is do you want it at your scale? A lot of companies are actually choosing to do it right now because they know they will grow tomorrow. (36:41)

Sahil: Keep in mind it is a managed offering. The benchmarks tell that Snowflake performance is essentially the same for Iceberg tables compared to standard Snowflake tables. Another challenge you face with Iceberg is that query performance is generally slower than a standard engine. There have been a lot of performance improvements and the difference was only ten to twenty percent in some benchmarks. If you are serving a team where wait times matter ten to twenty percent is still a lot. (37:19)

Alexey: There was another thing and I probably should let you finish before I ask the follow up. (38:20)

Sahil: Another thing is if you are getting a lot of streaming data it creates a lot of metadata files every time you commit. Sometimes your actual Parquet data files are smaller but the accumulated metadata is higher. That is not a good approach so we encourage balancing it out by using a Kafka queue and committing on a schedule. If it is a smaller data set and high frequency commits I would not recommend it. (38:27)

Alexey: In my case I would be better off just having a separate Postgres and doing analytics there. When I have ten million records in my fact tables then I can think about what makes sense for me to do later. (39:18)

Sahil: Absolutely. (39:35)

Alexey: I see a few questions and this is what I wanted to cover. You contribute to Apache projects like OSI. It was renamed to open semantic interchange. (39:37)

Sahil: I contributed to a few PRs. It is still incubating so it is not a full Apache project yet. (39:56)

Understanding Apache OSI (Open Semantic Interchange)

Alexey: Can you tell us more about this project? (40:11)

Sahil: Let us take an example where Alexey queries revenue for the last ten days. That is the only question Alexey asked but the definition could mean so many things. Does it include business days or not? Do I need to make sure the revenue includes tax or not? In technology several platforms consume the data but in the world of LLMs definitions may vary. (40:14)

Sahil: When a model queries your table you want to make sure the returned result is as deterministic as possible. It should not be fluctuating when queried differently. The core thesis here is defining how your semantic view should look. You are defining trusted queries and how specific questions should be answered. These are the dimensions the metrics and how calculations will be done on top of it. (41:03)

Sahil: These definitions can live in different engines but you want them to be portable. Today you have a semantic view defined in Snowflake but you want dbt or Databricks to use it. You do not want to create the same definition in Databricks. What OSI does is act as a code converter or transpiler. You put it in one file and it exports another file compatible on the other side while the definition stays the same. (42:00)

Alexey: If I use multiple compute engines like Snowflake Databricks and BigQuery I can have one centralized data catalog. Then I can have OSI sitting between my catalog and all these engines. (42:35)

Sahil: Exactly. I think I did not mention that Iceberg is the physical layer while the logical layer still stays with you. If you were using Postgres only Postgres knows how that table looks through create table or view statements. Those statements are not portable to any other engine. To port it you create that semantic view essentially and it maintains the logical layer. (43:06)

Alexey: Where I was going with this example is having one central catalog and different compute engines that can access it. Does Apache Iceberg not already offer interoperability out of the box? Why do I need OSI on top of that? (43:36)

Sahil: These are two different things. Iceberg is a table format that simply defines how the table should look. On top of it sits the catalog which handles governance and tells how to access the table files and latest states. Catalogs can be many things like Glue or Nessie which are compatible with Iceberg. Iceberg is the storage layer and the catalog is about access. (44:00)

Sahil: The logical layer like a semantic view was not portable because definitions could vary between systems. What OSI is doing at a very simple level is letting you maintain the business logic of your code from one system to another. (44:41)

Alexey: What exactly do you mean by business logic? Is it our dbt transformations? (45:13)

Sahil: Let us not take the dbt transformation example as it may not be right. Let us come back to the original example where you select top ten customers from the last ten days. How do you qualify those top ten customers? That is the only question. (45:22)

Alexey: You mean select top ten customers is the only thing you give to the system. You are not actually writing a full select statement. You need a definition of what revenue is and how exactly you convert this query into an actual SQL query that gets executed. (45:48)

Sahil: Exactly. From an LLM perspective it does not know what top ten means or how to calculate it. The way you start is by giving some example queries to it in a semantic view. In that semantic view you define a set of queries that will get asked often by the agents. You also specify that a column name can be a synonym like first name or contact for phone number. (46:13)

Sahil: You provide a whole definition of how your data structure looks. That logic previously only lived in Snowflake. If you wanted to do the same thing on Databricks you had to write the same logic in Databricks SQL. You were doing that operation twice. OSI converts it directly so you just port it and use it there. It reduces a lot of friction for teams so your logical definition stays constant. (46:52)

Alexey: Just to make sure I understood correctly let us say the use case is ad hoc analytics. Top management opens this application and writes give me revenue for the last ten days for top ten customers. OSI takes this query and maps what top ten and revenue mean. It converts this natural English query into some logical query. This query will be converted to the specific SQL dialect depending on where the data is located like Postgres or Snowflake. We have this intermediate representation that is convertible to other things. (47:47)

Sahil: It is convertible to other things. You still have to create that one semantic view that can be ported. That is the key point. (48:50)

Managing Schema Changes in Data Warehouses

Alexey: What is this other thing you are working on? Was it Schema Change? (49:00)

Sahil: As we all know data evolves. A lot of teams had issues when they created an additional column in a table but had no way to know when or how it changed. Previously if I issued an alter statement to a database it ran through but there was no way to track it. Schema Change solves several things including providing a Git like structure. You can manage within GitHub how the file version changed and who made those edits. (49:05)

Sahil: It uses imperative coding to manage versions. Simply put you are maintaining version files and telling the system you previously had ten columns and now added an eleventh. There are files that you always run and files you want to repeat. It allows you to manage schema changes easily on Snowflake similar to technologies like Flyway or Liquibase. (49:49)

Alexey: I use Django and in Django we have migrations. When you were describing it I realized it is just like Django migrations but for warehouses. (50:32)

Sahil: Django is kind of a batteries included thing. If you use something like FastAPI or Flask they do not have migrations built in so you would use Liquibase. I am not a Django expert so I will not talk a lot about that. (50:45)

Alexey: Django has been the choice for me to develop web applications for the last sixteen years. It has been around for quite some time and is something I know quite well. Did you start this project or did you inherit it? (51:08)

Sahil: I did not start this project. I was at Slalom working with a top electronic manufacturer and they were using Schema Change. I worked there for one and a half years and kept using Schema Change for other projects because it was so powerful. It helped us manage how data evolved for auditability which made me very interested. I interacted with the authors and started contributing. (51:31)

Sahil: I am working there right now. If your audience wants to contribute or has any ideas let us know. We are happy to review and work through those. (52:24)

Alexey: I looked it up and right now it is under the Snowflake Labs organization. Has it always been under Snowflake Labs when you started using it? (52:47)

Sahil: The reason is it mostly works entirely on open source Snowflake. It is very much inspired by Flyway which can be used with multiple technologies. Snowflake also provides Database Change Management which is relatively the same but more declarative in nature. Schema Change is more imperative. It started under a person's name but eventually moved under Snowflake Labs. (52:54)

Alexey: We do not have a lot of time left but I am interested in your journey from a user to maintainer. You briefly mentioned you used it for a year and a half and then became a contributor and eventually a maintainer. Can you tell us more about this journey? (53:40)

Sahil: Is this for the audience to learn how they can become better contributors? (54:12)

How to Contribute to Open Source Projects

Alexey: Yes because I think your story is motivating. In DataTalks.Club we are big supporters of open source and want to give as much spotlight to open source tools as possible. I contribute a lot to open source and want people who listen to this to also do that. Perhaps your story will motivate somebody to become a contributor first and then eventually a maintainer. (54:20)

Sahil: Context really matters. When you start with open source contributions it feels like nothing because your pull requests might not get merged or your issues handled. A lot of people in open source are spending their own time outside of their full time job. You need to start working on a technology like Schema Change and you will automatically find scenarios that are not working. You have two options either make it a community wide project or fork it and deploy it in your own environment. (54:45)

Sahil: My story and the way I recommend others start is by looking at first good issues. Any open source project will have simple issues that anyone can start with. Start contributing to the developer mailing list. Data engineering is very much open source whether it is Apache NiFi for ingestion Spark for transformations Iceberg for tables or Kafka for streaming. Go through those and look at what entices you and fits your use case. (55:41)

Sahil: Start communicating on the developer list to understand how teams make decisions. Contribute to proposals and start creating pull requests to build trust. Once you build trust the community will vote if they want to make you a committer. Once they make you a committer you go on to mentoring other communities. For example OSI needs mentors because it is incubating. (56:34)

Sahil: You get a lot of visibility from your work. It takes a lot of time but it is motivating too. (57:21)

Alexey: You managed to nail it perfectly within the time constraint. Do you have time for one more little question? (57:29)

Sahil: Absolutely. (57:36)

Alexey: As a maintainer I am curious because sometimes people ask their AI assistant to fix an issue on open source. Not many maintainers welcome these kinds of automated contributions. What is your opinion on automated pull requests? (57:36)

Sahil: This is a very good point that you bring up. I have noticed in Iceberg and the OSI community that people just want to become a contributor by asking code agents to write something and post it. You need to ask yourself why you are contributing and if you are solving real issues. All the communities I have seen are welcoming AI contributions but there should be material behind it. You need to own and understand what you are writing. (58:08)

Sahil: I have seen original authors pushing back on things written using AI code. You need to be very thorough. These committers are investing their time to review your pull requests. If it does not bring substance and you just overload them with pull requests it shows you as a weak candidate. People should remember open source is truly open and people can see what you committed. (59:04)

Sahil: If you are doing bad things it is not good for you. I ask people to be very thoughtful in what they contribute. I am very much in favor of asking AI to evaluate what you wrote or to suggest how to author a solution. I think AI systems have made us a little bit dependent and we do not use our own minds as much. Use AI to your advantage but do not use it to your disadvantage. (59:53)

The Role of AI in Open Source Contributions

Alexey: Maintainers spend their time reviewing these requests so you want to show an equal level of commitment. Do not just open an AI tool and tell it to solve whatever open issue it is. Ideally it should be a problem you understand exactly and are actively engaged in. (1:00:40)

Sahil: Exactly. Sometimes someone will open a pull request that makes logical sense but the community previously decided it is not a right candidate for the next release. For interoperability you want other engines to comply with it and you know some will lag behind so you make those decisions. People need to really understand the whole ecosystem. You need to understand the developer list and what the community is truly trying to deliver. (1:01:06)

Sahil: Use AI only to your advantage and maintain that context. Do not completely delegate context management to AI. (1:01:46)

Alexey: Sahil thanks a lot for a wonderful discussion. Thanks for explaining many things to me. I have a clearer picture now because my understanding of modern lakehouses was quite outdated from the Glue and Hive times. Thanks for joining us today and I see that the audience also enjoyed our conversation. (1:02:01)

Sahil: Thank you Alexey it was great chatting with you. (1:02:34)

Alexey: Have a great day ahead. I know it is early for you so the early bird gets the worm. (1:02:36)