Cache vs. Database: Has Performance Converged?
Although caches and databases are different animals, databases have always cached data and caches are starting to use disks. Is their performance converging over time?
In-memory caches have long been regarded as one of the fastest infrastructure components around. Yet, it’s been a few years since caching solutions started to look into the realm of flash disks. These initiatives obviously pose an interesting question: If an in-memory cache can rely on flash storage, then why can’t a persistent database also work as a cache?
To quantify the strengths and tradeoffs of each, ScyllaDB joined forces with memcached’s maintainer to compare both across different scenarios.
We’ll explain our motivation for these tests, provide a summary of the tested scenarios and results, then present recommendations for anyone who might be deciding between ScyllaDB and memcached. Along the way, we analyze the architectural differences behind these two solutions and discuss the trade-offs involved in each.
Spoiler: the results reveal that:
- Both memcached and ScyllaDB maximized disks and network bandwidth while being stressed under similar conditions, sustaining similar performance overall.
- While ScyllaDB required data modeling changes to fully saturate the network throughput, memcached required additional IO threads to saturate disk I/O.
- Although ScyllaDB showed better latencies when compared to memcached pipelined requests to disk, memcached latencies were better for individual requests.
There’s also a detailed Gitbook for this project, with a more extensive look at the tests and results and links to the specific configurations you can use to perform the tests yourself.
ScyllaDB would like to acknowledge memcached maintainer Alan Kasindorf (a.k.a. dormando) and Danny Kopping for their contributions to this project, as well as thank them for their support and patience. Bonus: dormando recently joined me for a P99 CONF talk: “Why Databases Cache, but Caches Go to Disk.” You can watch that talk, plus hundreds of others (including a great one by Kopping) at https://www.p99conf.io/
RAM Caching Efficiency
The more items you can fit into RAM, the better your chance of getting cache hits. More cache hits result in significantly faster access than going to disk. Ultimately, that improves latency.
This project began by measuring how many items we could store to each data store. Throughout our tests, the key was between 4 to 12 bytes (key0 .. keyN) for memcached, and 12 bytes for ScyllaDB. The value was fixed to 1,000 bytes.
Memcached
Memcached stored roughly 101 million items until eviction started. It’s memory efficient. Out of memcached’s 114 gigabytes assigned memory, this is approximately 101G worth of values, without considering the key size and other flags:
Memcached stored 101 million items in memory before evictions started.
ScyllaDB
ScyllaDB stored between 60 to 61 million items before evictions started. This is no surprise, given that its protocol requires more data to be stored as part of a write (such as the write timestamp since epoch, row liveness, etc). ScyllaDB also persists data to disk as you go, which means that Bloom filters (and optionally indexes) need to be stored in memory for subsequent disk lookups.