Hey :) I would say it depends on what the company does. If you’re in the business of processing large amounts of data (e.g. you’re crawling large numbers of websites), it’ll be worth thinking about architecture up-front.
But most startups (e.g. typical SaaS apps) don’t initially have a lot of data. For these early stage companies, I would specifically advise against investing in a scalable architecture until you actually need it. The priority of an early-stage business is to be flexible and quickly adapt to the needs of customers as they are discovered. A complicated architecture is actually harmful here, because it reduces flexibility. Personally I’d just stick with PostgreSQL in this situation, and keep everything as simple as possible.

What exactly do you find unclear or do you want to know more about?

Data in memory typically has a different representation from data on disk. For example, in memory you can use pointers, but pointers to memory addresses don’t make sense on disk, since after you restart a computer all the data will be at different memory addresses.

Therefore, you need to convert between the disk-oriented representation and the in-memory representation. This conversion incurs a cost. A database that doesn’t maintain on-disk data structures can save the conversion, because it can work with the in-memory representation only.

when reading this from the book it gave me the impression is disk-based db + cache would be just as equivalent good and therefore we do not need to have in-memory db at all

or if we do then in-memory dbs essentially shall be used as cache.
I wonder if I miss any point if I think in that extreme?

For a more detailed argument I suggest reading the papers referenced in this section:
https://hstore.cs.brown.edu/papers/hstore-lookingglass.pdf
http://www.vldb.org/pvldb/vol6/p1942-debrabant.pdf

ok thanks Martin. let me check them out. I read the lookingglass one but not yet the other one. I probably misunderstood the whole thing

I’m afraid I don’t have enough first-hand experience of different companies to give a good answer.

I think whether a team is already familiar with a tool is an important factor. The fewer new tools you have to learn, the better! But it shouldn’t be the only factor: if you try to bend a single tool to all possible applications, you may run into difficulties too. As always, it’s a judgement call.

In practice, I suspect a lot of decisions are also made based on whether a technology is currently fashionable or not (e.g. whether the team members recently heard a conference talk about it, or read a tweet about it). That is poor practice in my opinion, but I can’t deny that it happens!

Cloud services reduce the amount of investment you need to make into operations, because someone else is providing the operations team for you. But in order to know how to use a cloud service effectively, you still need a good understanding of what it can and cannot do. And you often don’t get that understanding from reading the vendor’s documentation alone, because a vendor will always pretend that their product is great for all purposes, even if it has strengths and weaknesses.
The purpose of my book is to teach you the fundamentals so that you can figure out the strengths and weaknesses of different technologies. That applies equally, regardless of whether you’re self-hosting or using a cloud service.

Hi Slim! I have never worked at a bank, so I don’t have personal experience. But my impression is that it doesn’t make sense to talk about the suitability of a technology for banking as a whole, since banking is a complex business where different parts of the business may have very different requirements. The needs of the people doing fraud prevention or anti-money-laundering will likely be very different from the needs of the people processing mortgage applications. The needs of those dealing with investments or trading will be different again.

I understand if there is a preference of relational databases since they are quite versatile and can satisfy the needs of a fairly wide range of applications (though not everything), and sometimes it’s better to stick with a tool you are deeply familiar with (“the devil you know”), even if it’s not quite optimal for the use case, than to incur the cost of learning how to use and reliably operate a different tool.

Keep in mind that choosing to adopt a certain database is not just a matter of the developers choosing one API or query language over another. There is also the need to set up backups, disaster recovery, to have ETL to bring the data from operational systems into a data warehouse, maybe audit logs, etc. etc. — adopting a new data storage technology also requires figuring out all this infrastructure.

So I have a lot of sympathy for organisations that are hesitant to adopt new technologies, even if they are better suited to the problem than whatever they are currently using.

thanks for this detailed answer :relaxed::relaxed:

I’m afraid I don’t have a good pithy answer for this. Different companies/situations have different needs and different problems. Part of the ethos of the book is that there isn’t one single technology or principle that is universally applicable; rather, there are trade-offs, and knowing what they are will let you choose the right tool for the job.

I think the biggest challenge is perhaps that there is such a bewilderingly large number of tools out there, and it can be really difficult to figure out which is best for a given task. Hence a primary goal of the book is to help you figure out what questions you need to ask in order to choose an appropriate technology.

most microservices architecture use transactions only within the bounds of a single service, and don’t have distributed transactions across service boundaries. this can sometimes make it really difficult to keep data consistent. there have been attempts to allow cross-service transactions (such as WS-AtomicTransaction in the SOAP world) but I fear this is not a good answer, since it introduces tight coupling between services.

one approach I’ve seen recommended is to draw the boundaries between services such that cross-service transactions simply are not necessary. this means that if you have two services that do need atomic commit, you need to merge them into a single service — i.e. don’t make the services too fine-grained.

another idea I’ve seen is to use a persistent log such as Kafka for communicating between services (instead of just ephemeral HTTP/RPC requests). this doesn’t quite give you atomic transactions, but it does give you pretty strong assurance that if one consumer reads a message off the log, all of the consumers will read that message. you can build some pretty strong guarantees based on that.

I elaborate on that second idea in this article on Online Event Processing: https://martin.kleppmann.com/papers/olep-cacm.pdf
and Ben Stopford has also written about using Kafka for inter-service communication: https://www.confluent.io/designing-event-driven-systems/

thanks Martin. I have seen a similair approach mentioned in Microservices patterns book written by Chris Richardson.

Big question! I have been collecting ideas for a second edition but have not yet started on it. Some ideas for improvement are:

• talk more about hosted cloud DB-as-a-service infrastructure, which has gained popularity rapidly (but which still requires a lot of technical knowledge to use well)
  
• there are tons of new tools in areas such as time series, workflow engines, new transaction processing engines (“NewSQL”)

• maybe a bit more on decentralised and local-first technologies, since interest in those has been growing
  
• I was thinking it might be cool to interview people at a couple of major internet companies and get them to share examples of their architectures, as case studies to discuss in the book

I am collecting ideas for a second edition, but not planning to start work on it for at least another year, since the current content is still pretty up-to-date.
I replied about ideas for a second edition in this thread

some of this has happened, but my sense is that the big idea — redesigning interfaces and APIs around dataflow rather than request/response interaction — is still a long way away. there are plenty of REST APIs for online services out there, but not many of them let you subscribe to a log of changes. the best you can usually hope for is a webhook, and this is really just a notification mechanism — you can’t use it to reliably maintain derived views onto the data in someone else’s API.

Thank you for the answer!
But do you still believe this can/will happen, or “application” and “data” worlds are doomed to live apart :) (similar to DBMS and OS evolved separately)?

hard to say without a crystal ball. there is a huge amount of existing investment in REST API, and hence a huge inertia preventing a move to anything else. on the other hand, change data capture in databases seems to be becoming mainstream, and when you combine that with stream processing and event subscriptions you’re not too far off.

Linearizability is a very useful property and it makes sense to talk about it. However, it’s a property of a particular operation or set of operations; a system may well provide some operations that are linearizable and others that are not. For this reason it doesn’t really make sense to label the system as “consistent” or not in the sense of CAP. Rather, we should say, for example: the enqueue and dequeue operations are linearisable.

Defining the “strongest level of consistency we can have” is a bit tricky because there is no good formal definition of what “stronger” means. But I do think that in practice, linearizable enqueues/dequeues are probably the strongest useful consistency model you will find for a queue.

observable to whom? to another client that is querying A and B? atomic commit/2PC does not guarantee anything about concurrency (it’s about handling crashes cleanly, not about concurrency). it is entirely possible for a client to query A and B while the 2PC protocol is in progress, and to get different responses from the two nodes.

This would make for an interesting research project! I don’t think it’s possible to say for certain whether this approach makes sense without actually trying it and seeing whether it helps and where it breaks down.

There is a long-standing debate between implementors of database engines and implementors of operating systems about caching of disk pages in memory. The OS automatically uses memory that would otherwise be unused to cache recently accessed disk blocks. But then a DB may need its own buffer cache (e.g. because it needs to control when a dirty page is written back to disk, because it has to go to the WAL before the data page is written), so data ends up being cached twice, which wastes memory.

there is a difference between memcache/redis and these block-level caches, which is that application-level caches such as memcache/redis don’t just store disk blocks, but they store entire precomputed responses to complex queries, potentially including some business logic. serving that data from a cache doesn’t only save on I/O, but also on computation time, which might be significant.

Cool. Thank you very much for your reply. Just want to add that Oracle has its own ROW cache. “When a query executes, the database searches the cache memory to determine whether the result exists in the result cache. If the result exists, then the database retrieves the result from memory instead of executing the query. If the result is not cached, then the database executes the query, returns the result as output, and stores the result in the result cache.
When users execute queries and functions repeatedly, the database retrieves rows from the cache, decreasing response time. Cached results become invalid when data in dependent database objects is modified.”

I wouldn’t necessarily say that “LST writes less to disk” (a statement about write amplification), but rather that LST writes tend to be a more sequential access pattern compared to B-tree writes, and so there can be a performance advantage on disks where random-access writes are slower than sequential ones.

as far as I know, a disk always writes data an entire block at a time, even if you only change 1 byte in a file. (that’s why linux calls a disk a “block device”.)

a relational DB will tend to have its own concept of pages that doesn’t map exactly to physical disk blocks, but is roughly related.

it’s not the case that every database block/page maps to a separate file (file systems would not cope well with such a large number of small files). I think DBs usually allocate one big file containing many blocks, and use offsets into that file to identify blocks.

if you update just one row, you’re right that the WAL write will happen first, but the page containing the row will also have to be written sooner or later, even if the transaction is allowed to commit before it is written.

it will depend on the specific system, but in general I don’t see any reason why you can’t have one client to connect to multiple leaders. one potential setup would be for a client to connect to a leader in the local DC by default, and fall back to connecting to a different DC in the case of problems.

I think it’s excellent, and more systems besides PostgreSQL should use it!

Perhaps, although other authors have already covered streaming systems in great detail, so I’m not sure there is much more for me to add.

Just to follow-up on this: which authors would you recommend reading to dive deeper into streaming systems?

“Streaming Systems” by Akidau et al.; “Kafka - The definitive guide” by Narkhede et al.

(I think a second edition of the Kafka book is in the works)

Thank you!

I got tired of the short-termism in industry, especially in startups, where everything has always got to happen “right now”. I wanted a setup where I would have the space to think, to take the time to really understand things, and to work to improve the foundations of how we write software. In research I can work on things that may not be practical for another 5–10 years, and that’s fine. In a company you can’t normally work with such a long time horizon.

I can totally relate to that. Thank you for your answer!

I’m afraid this is not my area; others are working very actively on this, but it’s fast-moving and I am not up-to-date. I suggest looking up work by folks such as Timnit Gebru and Cathy O’Neil.

This is the only multi-lecture course available so far. I also have this playlist of conference talks I have done: https://www.youtube.com/watch?v=fU9hR3kiOK0

it includes some course-like material, such as this 2-hour lecture on formally verifying distributed algorithms: https://www.youtube.com/watch?v=Uav5jWHNghY

q1: it’s difficult to make a concrete recommendation without knowing much more about the characteristics of your data, the access patterns and queries you use, etc. You might be able to continue using Postgres by splitting the database across multiple machines, perhaps using something like Citus. If you don’t do many joins in queries, you could consider using a key-value store/document DB such as Cassandra or MongoDB, but that would be a much bigger change.

a distributed filesystem or an object store like S3 would make sense if the data items are large. this would scale well, but would make it much harder to do any indexing (i.e. finding documents based on some value that occurs within a document).

q2: Saving data in raw form, and separately storing data derived from it, is a great pattern. If you want to change your processing logic, you can then re-run it on the raw data. The datastore to use for the derived data will again depend on the data format and your access patterns.

q3: my suggestion would be knowing how to reason about trade-offs. there is never one true answer, only various options that each have their pros and cons. being able to figure out those strengths and weaknesses, and communicating them to the rest of the team, seems very valuable to me.

Thank you so much for responding to my questions.

Thanks! I am considering ideas for a book that would go into more details of algorithms used in distributed systems. But work has not yet started, so there is no timeline for when this might happen.

I think programming languages are a question on which people have very strong opinions because each language is like a tribe, and people couple their identity to the tribe they belong to. In my opinion, the language in which a system is written rarely makes a big difference; more important is the system architecture. Most mainstream languages are probably okay for building many types of systems.

In particular, I think questions of expressiveness and details of language features do not have a big effect when it comes to system operations. What does make a difference is a language’s runtime characteristics, such as whether it supports threads, and whether it uses a garbage collection runtime. No GC means no GC pauses, which can be important for low-latency systems.

Rust is interesting to me because it’s memory-safe, supports threads, has no GC runtime, and is very portable (you can use it to build mobile apps or servers or web apps by compiling to wasm). But in the end, it’s the needs of a particular system that matter. There is no one language that is perfect for all systems programming.

I am a big believer in CRDTs, and have been doing research on them for the last 6 years! I maintain a community website https://crdt.tech/ that has a lot of resources on CRDTs, and links to all the latest research (including my own).

Well, this is going to be very subjective, since every person has their own priorities! Personally I am excited about the potential of local-first software, and moving away some of the current cloud-centric view of distributed systems. That’s what I’m going to be working on for the next 5 years, anyway!

Ah yes, I have struggled with this as well. I’m afraid a proper explanation goes beyond what I can provide here in a few sentences. I should do a blog post on this at some point.

at a high level, the difference is in purpose. a vector clock is used to compare events in a distributed system, and figure out which happened before which, and which are concurrent. a version vector is used to compare states of replicas, and figure out whether one state supersedes the other. the mechanism in both cases is similar: a vector of numbers that are incremented.