ScyllaDB Hits Fourth Generation with Raft, Tablets, and a Cloud-First Vision

In business, they say it takes ten years to become an overnight success. In technology, they say it takes ten years to build a file system. ScyllaDB is in the technology business, offering a distributed NoSQL database that is monstrously fast and scalable. It turns out that it also takes ten years or more to build a successful database.

This is something that Felipe Mendes and Guilherme Nogueira know well. Mendes and Nogueira are Technical Directors at ScyllaDB, working directly on the product as well as consulting clients. Recently, they presented some of the things they’ve been working on at ScyllaDB’s Monster Scale Summit, and they shared their insights in an exclusive fireside chat.

https://www.youtube.com/watch?v=3cFNp3tpRdw&embedable=true

You can also catch the podcast on Apple,  Spotify, and Amazon

The evolution of ScyllaDB

When ScyllaDB started out, it was all about raw performance. The goal was to be “the fastest NoSQL database available in the market, and we did that – we still are” as Mendes put it. However, as he added, raw speed alone does not necessarily make a good database.

Features such as materialized views, secondary indexes, and integrations with third party solutions are really important as well. Adding such features marked the second generation in ScyllaDB’s evolution. ScyllaDB started as a performance-oriented alternative to Cassandra, so inevitably, evolution meant feature parity with Cassandra.

The third generation of ScyllaDB was marked by the move to the cloud. ScyllaDB Cloud was introduced in 2019, has been growing at 200% YoY. As Nogueira shared, even today there are daily signups of new users ready to try the oddly-named database that’s used by companies such as Discord, Medium, and Tripadvisor, all of which the duo works with.

The next generation brought a radical break from what Mendes called the inefficiencies in Cassandra, which involved introducing the Raft protocol for node coordination. Now ScyllaDB is moving to a new generation, by implementing what Mendes and Nogueira referred to as hallmark features: strong consistency and tablets.

Strong consistency and tablets

The combination of the new Raft and Tablets features enables clusters to scale up in seconds because it enables nodes to join in parallel, as opposed to sequentially which was the case for the Gossip protocol in Cassandra (which ScyllaDB also relied on originally). But it’s not just adding nodes that’s improved, it’s also removing nodes.When a node goes down for maintenance, for example, ScyllaDB’s strong consistency support means that the rest of the nodes in the cluster will be immediately aware. By contrast, in the previously supported regime of eventual consistency via a gossip protocol, it could take such updates a while to propagate.

Using Raft means transitioning to a state machine mechanism, as Mendes noted. A node leader is appointed, so when a change occurs in the cluster, the state machine is updated and the change is immediately propagated.

Raft is used to propagate updates consistently at every step of a topology change. It also allows for parallel topology updates, such as adding multiple nodes at once. This was not possible under the gossip-based approach.

And this is where tablets come in. With tablets, instead of having one single leader per cluster, there is one leader per tablet. A tablet is a logical abstraction that partitions data in tables into smaller fragments. Tablets are load-balanced after new nodes join, ensuring consistent distribution across the cluster. Any changes to Tablets ownership are also ensured to be consistent by using Raft to propagate these changes.

Each tablet is independent from the rest, which means that ScyllaDB with Raft can move them to other nodes on demand atomically and in a strongly consistent way as workloads grow or shrink.

Speed, economy, elasticity

By breaking down tables into smaller and more manageable units, data can be moved between nodes in a cluster much faster. This means that clusters can be scaled up rapidly, as Mendes demonstrated. When new nodes join a cluster, the data is redistributed in minutes rather than hours, which was the case previously (and is still the case with alternatives like Cassandra).

When we’re talking about machines that have higher capacity, that also means that they have a higher storage density to be used, as Mendes noted. Tablets balance out in a way that utilizes storage capacity evenly, so all nodes in the cluster will have a similar utilization rate.

That’s because the number of tablets at each node is determined according to the number of CPUs, which is always tied to storage in cloud nodes. In this sense, as storage utilization is more flexible and the cluster can scale faster, it also allows users to run at a much higher storage utilization rate.

A typical storage utilization rate, Mendes said, is 50% to 60%. ScyllaDB aims to run at up to 90% storage utilization. That’s because tablets and cloud automations enable ScyllaDB Cloud to rapidly scale the cluster once those storage thresholds are exceeded, as ScyllaDB’s benchmarking shows.

Going from 60% to 90% storage utilization means an extra 30% per node disk space can be utilized. At scale, that translates to significant savings for users. Further to scaling speed and economy, there is an additional benefit to tablets: enabling the elasticity of cloud operations for cloud deployments, without the complexity.

Something old, something new, something borrowed, something blue

Beyond strong consistency and tablets, there is a wide range of new features and improvements that the ScyllaDB team is working on. Some of these, such as support for S3 object storage, are efforts that are ongoing. Besides offering users choice, as well as a way to economize even further on storage, object storage support could also serve resilience.

Other features, such as workload prioritization or the Alternator DynamoDB-compatible API, have been there for a while but are being improved and re-emphasized. As Mendes shared, when running a variety of workloads, it’s very hard for the database to know which is which and how to prioritize. Workload prioritization enables users to characterize and prioritize workloads, assigning appropriate service levels to each.

Last but not least, ScyllaDB is also adding vector capabilities to the database engine. Vector data types, data structures, and query capabilities have been implemented and are being benchmarked. Initial results show great promise, even outperforming pure-play vector databases. This will eventually become a core feature, supported on both on-premise and cloud offerings.

Once again, ScyllaDB is keeping with the times in its own characteristic way. As Mendes and Nogueira noted, there are many ScyllaDB clients using ScyllaDB to power AI workloads, some of them like Clearview AI sharing their stories. Nevertheless, ScyllaDB remains focused on database fundamentals, taking calculated steps in the spirit of continuous improvement that has become its trademark.

After all, why change something that’s so deeply ingrained in the organization’s culture, is working well for them and appreciated by the ones who matter most – users?