Despite its importance in multiple industries, time series data continues to be one of the more challenging datasets to handle at scale. Time series help us identify trends in data, letting us demonstrate concretely what happened in the past and make informed estimates about what will happen in the future.

But for TIMESLIDERS, time series data was the backbone of their business, but also the bane of their users. TIMESLIDERS is a mobile-application performance library, built for developers. Their client application would monitor as many performance details about the client device and application as possible, reporting things back like avg network speed, quality of wifi connection, battery life, foreground activity paint rate, etc, etc.

Clients could then log into a portal and see a plethora of performance statics on how their app was performing around the world, on real devices.

Sadly, this process of building and displaying these performance tables quickly became a bottleneck, and a less-than-ideal experience for the majority of their clients.

Thankfully, we didn’t have to look far to find the problem : In order to build their graphs, the very first thing they did was query by timestamp.

Many schema designers are inclined to define a root table that is timestamp ordered, and updated on every write. Unfortunately, this is one of the least scalable things that you can do with Cloud Bigtable.

What’s going wrong?

Let’s be clear: Storing time-series data in Cloud Bigtable is a natural fit. Cloud Bigtable stores data as unstructured columns in rows; each row has a row key, and row keys are sorted lexicographically.

There are two commonly used ways to retrieve data from Cloud Bigtable:

• You can get a single row by specifying the row key.
• You can get multiple rows by specifying a range of row keys.

These methods are ideal for querying time-series data, since you often want data for a given time range (for example, all of the market data for the day, or server CPU statistics for the last 15 minutes). As a result, Cloud Bigtable is functionally a great fit for time series.

Of course, there’s always a devil in the details, and for time series data, this is no exception. In brief, when a row key for a time series includes a timestamp, all of your writes will target a single node; fill that node; and then move onto the next node in the cluster, resulting in hotspotting.

Properly handling time series data

Let’s take a simple example from TIMESLIDERS: Storing battery status. Every 100ms or so, their app would log the battery status, creating an entry into Bigtable where the row key consists of the word “BATTERY” plus a timestamp. (for example: BATTERY20150501124501003).

The problem with this, is that the row key will always increase in sequence, meaning every row will have a high level of adjacency to every other row; And because Cloud Bigtable stores adjacent row keys on the same server node, all writes will focus only on one node until that node is full, at which point writes will move to the next node in the cluster. During this time, you end up with a large level of hotspotting.

So we need the ability to query a set of timestamped rows for our graph, but need to find a way to do it that doesn’t result in hotspotting.

Thankfully, there are a few ways to solve this problem.

Field promotion involves moving fields from the column data into the row key to make writes non-contiguous. For example, TIMESLIDERS could promote the USERID from a column to an element of the row key. This change would solve the hotspotting issue because user identifiers will provide a more uniform distribution of row keys. As a result, writes will be split across multiple nodes in their cluster.

The advantage of field promotion is that it often makes your queries more efficient as well, making this strategy a clear winner. The (slight) disadvantage is that your queries are constrained by your promoted fields, leading to rework if you don’t promote the right fields.

Another option is Salting, which adds an additional calculated element to the row key to artificially make writes non-contiguous. For example, take a hash of the timestamp and divide it by the number of nodes in the cluster (to help balance traffic) and append the remainder to the row key.

The advantage of salting is its simplicity — it’s essentially a simple hashing function that helps balance access across nodes.

On the other hand, it does require an extra step when you query for time ranges. You’ll have to do one scan per salt-value, and then recombine the results in your own code. Salting also makes it much harder to troubleshoot performance issues with Key Visualizer for Cloud Bigtable, because it’ll show row keys with a bunch of hashed values instead of human-readable strings.

The fix is in!

For TIMESLIDERS salting was problematic. Despite a barrage of tests, it was very difficult to choose a salt value that both distributes activity across nodes and still remained valid as their system scaled up and down.

Aa such, they decided to use field promotion, which allowed them to avoid hotspotting in almost all cases, and tended to make it easier to design a row key that facilitates queries.

Oh Hey!

It’s worth pointing out that time series data is a common enough challenge with Bigtable, that there exists a 3rd party library that can do all the heavy lifting for you. If you’re not interested in solving all these problems yourself, check out OpenTSDB which can save you a ton of headache later on.

From a performance perspective, one of the most important things to consider while designing your Bigtable Schema is how to properly design your index. And our friends at Urban Fitness Forge ran into this headlong recently.

Urban Fitness Forge is one of the latest in awesome real time fitness tracker products. Their advanced clothing line carries over 50 sensor measurements for the human body, each sampled 10 times a second. After their initial series A funding, they saw a huge boom in usage for users as they were picked up by a local clothing company. The result, too much data for their existing back-ends, which prompted them to move to the ever powerful Cloud Bigtable.

However, after their migration, they suddenly noticed that it was taking a lot of memory and time to fetch / sort results for their real time updating of statistics. By the time I got involved, their engineering team was in a Code Red to find the problem.

Finding the problem

UFF’s usage pattern for their data dominantly queried lat/long values to find specific sensors within an area for analysis. Or rather, given how often they ran this query, it was way too slow for their needs.

Their schema looked similar to what you’d expect. Their key was comprised of long/lat values for a sample inserted with a delimiter in the middle (e.g. “33.749#-84.388”) :

Performance wise, this simple design was creating a massive bottleneck for them.

While looking at some logging data, we quickly realized that our scans for data points near Portland, Oregon (USA) ended up also scanning and filtering through data points in Milan & Venice Italy. After some digging, researching, and asking around, we realized that there were three problems working in unison to cause this performance issue:

1. These cities are very close to each other in terms of Latitude:
2. Bigtable stories entries in their shards lexicographically by their index
3. Bigtable doesn’t have explicit support for secondary indexing.

Let’s break this down a little more.

Bigtable stories entries in their shards lexicographically by their index, meaning that entries with lexicographically similar keys will be closer to each other in the tablets. Queries that access a single row, or a contiguous range of rows, execute quickly and efficiently. However if need something like a secondary index (like the “LONG” in a “LAT/LONG” pair) then your queries end up in a full table scan, which will be far slower. A full table scan is exactly what it sounds like — every row of your table is examined in turn, which, as your table sizes grow, your performance suffers more.

This is where point #1 above comes into play. Portland is very close to two other large cities in terms of latitude:

The above image shows Portland, Milan and Venice overlaid on a map, you can see the full interactive map here.

As such, their storage in rows tends to be very close to each other, despite being geographically very far away. This means a full table scan involves walking through all the table entries that share a latitude value across the globe — A much larger scan size than we were hoping for.

Building the right index

This continues to highlight an important aspect of Bigtable schema design: choosing a row key that facilitates common queries is of paramount importance to the overall performance of the system.

For Urban Fitness Forge, that meant trying to create an index such that entities close to each other physically were sorted close to each other in table space, in order to reduce the scan size.

Thankfully, this has been solved before. A space filling curve is a way to organize a 2D space such that adjacent indexes are pseudo similar in physical space (I like this explanation if you are looking for more).

For example, if we stored points in a 2D set using a raster index (index = Y*WIDTH+X) we’d end up with something that looked like this:

Meanwhile, a space filling curve does this:

You can see that in the first example (aka RASTER order) that while 6 looks physically close to 2, it’s a whole row away from it, in terms of index. However, in the Space Filling Curve, the closest index values to 6 are physically close as well.

The result is a modified index for entities which use a space-filling curve integrated with a timestamp value to create a single, integer based index.

For Urban Fitness Forge, this meant a massive performance improvement for queries that sampled row data via lat/long, as it allowed items which were geographically close to each other, to also be adjacent to each other in the table.

Your mileage may vary

Now, obviously using a Hilbert space filling curve is not the solution to every Cloud Bigtable index option, but it does highlight the important part: to get the best performance out of Cloud Bigtable, it’s essential to think carefully about how you compose your row key. By choosing the correct row key template, you can avoid a painful data migration process later.

To do this, start by asking how you’ll use the data that you plan to store. For example:

• User information: Do you need quick access to information about connections between users (for example, whether user A follows user B)?
• User-generated content: If you show users a sample of social-style content, such as status updates, how will you decide which status updates to display to a given user?
• Time series data: Will you often need to retrieve the most recent N records, or records that fall within a certain time range? If you’re storing data for several kinds of events, will you need to filter based on the type of event?

By understanding your needs up front, you can ensure that your row key template, and your overall schema design, provide enough flexibility to query your data efficiently.

Cloud Bigtable : Getting the geography right

FlightMight was having trouble with Cloud Bigtable performance from their clients around the globe. Their system for monitoring aircraft information meant sampling lots of instrument data hundreds of times a second, submitting it all back to a server for storage and analysis.

Sadly though, their aircrafts tended to work all over the world, with shoddy connections and quickly changing latency situations, leaving their performance when interfacing with Cloud Bigtable, less than ideal.

Location, location location.

The issue here is that Cloud Bigtable, in order to be as performant as possible, is local to a specific region. As such, accessing Cloud Bigtable from clients around the world is quickly at the mercy of latency, which is why we recommend that any client accessing Bigtable be, not only in the same region, but specifically, in the same zone.

To show this off, here’s a small test (using a Cloud Bigtable loadtest tool written in Go) I ran where the client in us-central1-c connecting to cluster in:

Don’t get me wrong, 24ms/26ms 95th read/write percentile is extremely impressive, but we can see that almost doubles when we move the cluster to us-east4-a; and if you have a client that’s connection to Bigtable repeatedly, (say pushing up time-stamped samples) then this overhead starts to add up.

Bridging the gap.

Now, let’s be clear — This is not a challenge unique to Cloud Bigtable. The general challenge lies in the fact that we have a resource that’s geographically separated from our clients, and the latency overhead of repeated transactions can cause performance issues. Here on Cloud Performance Atlas, we’ve seen this same issue with respect to latency in Networking, Cloud Storage, Cloud Datastore and App Engine, so we’ve got a nice set of tools at our disposal.

Here are a few ideas we tossed around:

1) Use more clusters. From the Cloud Platform Console, if you create additional clusters, Cloud Bigtable will immediately start replicating data between those clusters (and we require the clusters to be in different zones). This is obviously the most straightforward way to solve this problem, as it allows your data to be replicated to geographically diverse locations. On the down side, this solution comes with extra cost, since you’ve got a second cluster running.

2) Update the clients to use multithreaded push/pull. This would mean that the overhead of latency wouldn’t be hurting the performance of the client, although we’d need to build their client robust enough to queue up samples, submit them in batches, and deal with situations where we lose connectivity. In this scenario, the latency is mitigated to the client’s perspective,but we end up with a lot of potential edge cases we might need to discover & code for. (imagine a scenario where push latency is ~80ms, and we’ve got a device that’s sampling information every ~10ms.)

3) Use Cloud Pub/Sub & a f1-micro instance. Cloud Pub/Sub is a fantastic deliver-once service for GCP. If we have our clients push their sample data into Cloud Pub/Sub, then we don’t need to have any overhead for buffering on the client. On the flip-side though, we would need to spin up a permanent Compute Engine VM instance, in the same zone, which pulls the messages from Cloud Pub/Sub, and then submits them to Bigtable directly. The challenge with this setup is multi-fold. Firstly, we’d need to use an f1-micro Compute Engine VM instance in order to keep costs down, which might not have the performance we need for connections (thus we’d need to scale, incurring higher costs). Secondly, this really only works for write operations from the client; reading data would need a separate path, which might involve different complexity to the endpoint.

4) Serverless endpoint. Cloud Functions and App Engine can help offer the positives of #1 / #2 above, without the downsides. For both reads & writes, Cloud Functions and App Engine can both scale to thousands of instances, each one properly connecting to the nodes in a Cloud Bigtable cluster (which, as we’ve talked about, are designed to handle 10k QPS of reads or writes per node if you’re using SSD storage). And placing the serverless systems in the same region as the Cloud Bigtable cluster means that latency, per-request can fall back inside of Google’s High-speed network, which can improve performance.

And for FlightMight, that was all they needed to hear. They adjusted their clients to push content to Cloud Pub/Sub, which then triggers a Cloud Function that’s deployed in the same zone as their Cloud Bigtable cluster, giving them the performance they need, and lowering their cost.

When you’re dealing with something as big and powerful as Cloud Bigtable, it’s important to make sure you’re looking at the right data and tests to ensure performance is awesome. As we talked about before, most of the magic of how Cloud Bigtable handles such scale is all behind the scenes; where frontends can coordinate, balance, and optimize your access patterns by moving tablets of data between various compute nodes as needed.

Because so much of the heavy lifting of Cloud Bigtable performance happens behind the scenes, it might not be obvious how to figure out whether an instance is performing well. Are you supposed to set a breakpoint in the middle of Cloud Bigtable’s tablet sharding or something? Do you even want to? (Spoiler alert: You don’t.)

Thankfully, Cloud Bigtable gives you a much easier way to see how your instances are performing over time.

Thankfully, this is where Cloud Bigtable Monitoring can help out.

The monitoring UI

Cloud Bigtable has a built in monitoring tool that automatically reports how your nodes, clusters, and instances are performing. After your instance is created, you can click the monitoring tab:

Which will show you things like CPU utilization, error rate, storage utilization, rows read/written read/write requests and read/write throughput; all in a handy graph form so you can see how these values have responded over a historic time series.

Now, having all this data is great, but what’s really important is finding the situations where it actually tells you something. Here’s a few common use cases that are worth keeping an eye on

Is it time to add more nodes?

As mentioned before, Cloud Bigtable works by keeping a set of front-end nodes which are responsible for routing and balancing traffic to the backend tables. When you create your Cloud Bigtable instances and clusters, you specify the number of nodes you want, and using the monitoring tool, you can quickly see if it’s time to add more nodes.

On our instances tab, we can take a look at the CPU utilization table, which has a “recommended max” line. This line represents the best practices for CPU utilization, and if you eclipse this line for a number of minutes, it’s generally a good idea to add more nodes to your cluster.

In addition, once the CPU utilization gets high enough, the nodes become bottlenecked, and we can see, as a result, that both the read throughput and wrote throughput suffer as a result.

On the non-cpu side, the “Disk Load” graph could also point to a need to add more nodes. If this value is frequently at 100%, you might experience increased latency. Add nodes to the cluster to reduce the disk load percentage.

How balanced are my reads/writes?

You might see another common performance problem with Cloud Bigtable if reads and writes aren’t evenly distributed across all of your data.

Despite designing your schema correctly, there are some cases where you can’t avoid accessing certain rows more frequently than others. Cloud Bigtable helps you deal with these cases by taking reads and writes into account when it balances tablets across nodes.

For example, suppose that 25% of reads are going to a small number of tablets within a cluster, and reads are spread evenly across all other tablets:

Cloud Bigtable will redistribute the existing tablets so that reads are spread as evenly as possible across the entire cluster:

However, there are times when Cloud Bigtable can’t properly spread out the read/write volume across all nodes, and as a result, performance will suffer.

We can detect this instance in the monitoring tool by keeping an eye on the “CPU Utilization of the hottest node”. If the hottest node is frequently above the recommended value, even when your average CPU utilization is reasonable, you might be accessing a small part of your data much more frequently than the rest of your data.

To fix this, you’ll need to go back and check your schema design, and your access patterns and make sure it supports a more even distribution of reads / writes across each table.

Digging deeper with Key Visualizer

The monitoring UI is a great way to figure out high level issues with your cluster, and see if there’s things you can address without too much heavy lifting. Often times though, you really need to dig into the specifics of your performance, and for that the Key Visualizer for Cloud Bigtable is exactly the tool for you, as it can provide insights into usage patterns at scale that are difficult to understand otherwise. For example:

• Check whether your reads or writes are creating hotspots on specific rows
• Find rows that contain too much data
• Look at whether your access patterns are balanced across all of the rows in a table

For an example from the official documentation, this tool can help you find specific usage patterns, visually, so you can quickly track them down:

There’s more to come!

Now, it’s worth noting that the values I’ve discussed above are for instances that don’t use replication. If you’re using replication, you should check out the official documentation to keep up-to-date with the best practices and performance thresholds.

If you’d rather monitor your Cloud Bigtable instances programmatically, we’ve got tools for that, too. You can use Stackdriver Monitoring to keep an eye on usage metrics, and Stackdriver Logging lets you audit access to your instances. If you’re using our HBase-compatible client library for Java, you can even get client-side performance metrics.

In upcoming posts we’ll be taking a look at how to properly design your schema, update hotspots, and design your data so you can run as fast as possible. Stay tuned!

When it comes to performance at scale, there’s very few products that can truly represent the “at scale” part the way that Cloud Bigtable does. To date, Bigtable is the technology behind the majority of Google products, including Gmail, Maps, Youtube; Each of which serves multi-billion active users.

What’s really amazing is that this same technology that powers Google’s most popular products is also available for you to use for your applications. But be warned : You’d think with all that scale and performance that it would be the silver bullet to handle all your backend needs, but when it comes to performance, just like anything else, there’s some things it’s good at, and some things it’s not.

How it works

As a massively oversimplified generalization, Cloud Bigtable can be described as three systems working together:

• A frontend server pool.
• A set of compute nodes to handle connections (aka a “Cloud Bigtable Cluster”).
• A scalable backend of storage.

In more detail, all client requests go through the front-end server pool before they are sent to a Cloud Bigtable node. Each node in the cluster handles a subset of the requests for the entire system, and the frontends will handle balancing these connections depending on the type of action the connection is interested in, and what part of your data it’s wanting to work on.

To store the underlying data for each of your tables, Cloud Bigtable shards the data into multiple tablets (Not a typo! Tablets and tables are different things.), where each tablet contains a contiguous range of rows within the table.

And here’s the important thing when it comes to tablets: they can be reassigned to different nodes in your cluster, on demand, allowing Cloud Bigtable to scale and re-balance seamlessly as your use patterns change.

For example, as reads/writes to a specific tablet increase, Cloud Bigtable might split a tablet into two or more smaller tablets, either to reduce a tablet’s size or to isolate hot rows within an existing tablet. Likewise, if one tablet’s rows are read extremely frequently, Cloud Bigtable might store that tablet on its own node, even though this causes some nodes to store more data than others.

Expected performance

Under these typical workloads, Cloud Bigtable delivers highly predictable performance, and according to the official documentation, you can expect to achieve the following performance for each node in your Cloud Bigtable cluster, depending on which type of storage your cluster uses:

(Standard disclaimer : These performance numbers are guidelines, not hard and fast rules. Per-node performance might vary based on your workload and the typical size of each row in your table. Please see here for latest numbers)

In general, a cluster’s performance increases linearly as you add nodes to the cluster. For example, if you create an SSD cluster with 10 nodes, the cluster can support up to 100,000 QPS for a typical read-only or write-only workload, assuming that each row contains 1 KB of data.

When to/not use it

From here, you can clearly see that Cloud Bigtable does not act like relational databases, and more to the point : while it scales amazingly well, and isolates hotspots gracefully, that doesn’t make it a perfect use for every scenario.

Cloud Bigtable excels at handling very large amounts of data (terabytes or petabytes) over a relatively long period of time (hours or days). This is because Cloud Bigtable needs time to learn your access patterns, and the data needs to be large enough to warrant usage of all nodes in your cluster (otherwise, you might get hotspotting to once cluster).

Which means that Cloud Bigtable is a great solution for:

• Time-series data, such as CPU and memory usage over time for multiple servers.
• Marketing data, such as purchase histories and customer preferences.
• Low Latency Serving, such as adtech (martech) recommendations, etc.
• Financial data, such as transaction histories, stock prices, and currency exchange rates.
• Internet of Things data, such as usage reports from energy meters and home appliances.
• Graph data, such as information about how users/machines/events are connected to one another.

On the other side, if you’re storing a small amount (< 300 GB) of data, or if your Cloud Bigtable interactions last for a very short period of time (seconds rather than minutes or hours), Cloud Bigtable won’t be able to balance your data in a way that gives you good performance, and as such, might not be the right technological solution for your application. Fret not! Google Cloud Platform has a great set of other offerings that might offer you what you need, for example:

• If you need relational joins, like for online transaction processing — use Cloud SQL
• If you’re using Blobs over 10MB — use Cloud Storage
• If you need ACID transactions — use Cloud Datastore or Cloud Spanner
• If you don’t have much data yet — use Cloud Datastore, the Firebase Realtime Database, or Cloud SQL

If your workload falls into the Cloud Bigtable space, then stay tuned to Cloud Performance Atlas, because we’ve got more content on Cloud Bigtable coming soon!

With the boom of mobile devices across the world, we’ve been seeing a lot of areas where smartphone adoption outpaced the ability for the telecom providers to update their networks. The result is a less-than-fast experience for the users, and a deep need to re-evaluate how our current internet protocols are working on the modern web.

This is some of the fundamental reasoning behind Google’s development of the QUIC protocol.

QUIC (Quick UDP Internet Connections) is a modern transport layer protocol that provides congestion control similar to TCP and security equivalent to SSL/TLS, along with improved performance (reduced connection and transport latency, higher throughput etc).

And recently the networking group in Google Cloud Platform released QUIC support for the HTTPS load balancers, which means it’s time to take it for a spin, and see what kind of performance we can get ;)

QUIC recap

A good way to decrease connection latency for an efficiently routed connection is to make fewer round-trips. Much of the work on QUIC is concentrated on reducing the number of round-trips required when establishing a new connection, including the handshake step, encryption setup, and initial data requests. QUIC clients would, for example, include the session negotiation information in the initial packet. This compression is enhanced by QUIC servers, which publish a static configuration record that is concisely referred to. The client also stores a synchronization cookie it received from the server, enabling subsequent connections to incur zero overhead latency (in the best case)

One of the motivations for developing QUIC was that in TCP the delay of a single packet induces head-of-line blocking for an entire set of SPDY streams; QUIC’s improved multiplexing support means that only one stream would pause

Originally developed back in 2012, the protocol works by using a set of multiplexed connections between two endpoints using UDP, focusing on SSL support, and reduced application latency.

Setting up the test

Step 1 : Enable QUIC on our HTTPS load balancer

If you’ve already got an HTTPs LoadBalancer up and running, enabling QUIC is very straightforward.

In the cloud console, simply go back to your Load Balancer, and enable the “QUIC negotiation” button in the frontend dialog.

Step 2 : Adjusting the network performance.

Much like BBR, QUIC works best in connectivity that’s challenged, so much like BBR we’ll allow our clients to simulate a bad network connection using a tc command:

Step 3 : the test

Our test is to fetch 10mb worth of files of various sizes from a GCS bucket through the load balancer, and chart the throughput. We’ll have one client connecting through an HTTP front-end (no QUIC support) with the other client connecting through QUIC.

The results

The results are pretty clear in this test, QUIC sees a 1.3x improvement in overall performance vs standard HTTPs on the load balancer, which is pretty nice for just setting one flag!

Where to use QUIC?

As mentioned, QUIC really shines in low-connectivity environments, where packet loss and delay are high. Much like BBR, there’s no downside to turning it on. Your QUIC enabled clients on high-performing networks will get a small boost, here and there, but you’ll see your biggest gains in the regions where connectivity needs some help.

Likewise, remember that your HTTPS Load Balancer is already getting the benefits of BBR for the Google Front Ends, so turning on QUIC just helps that even more!

Now it’s worth noting that QUIC requires a client which supports the protocol, so unless you’re writing your own client with support, make sure you check out what browsers support the protocol, to make sure you’re hitting your target market.

One of my favorite things about App Engine is that your requests are automatically traced, so you can see where during a request that your execution time is going. Shortly after, I showed a small technique that allowed you to insert custom trace blocks (since most trace data is RPC based).

Now, I’m really excited to talk about how tracing can be rolled out to all your other systems, even the ones not running on app engine.

Recap: Stackdriver Tracing

Stackdriver Trace allows you to analyze how customer requests propagate through your application, and is immensely useful for reducing latency and performing root cause analysis.

Trace continuously samples client requests, which it submits to the Stackdriver service to display their propagation / latency results.

One of the tricks though, is that it doesn’t fully capture your call graph (like a flame chart in Stackdriver Profiler would) but rather focuses on key paths around RPC calls, which generally result in most of your bottlenecks for the average application.

Everybody trace now!

As mentioned, if you’ve been an App Engine standard developer, chances are you’re familiar with this setup, since your requests have been getting traced automatically for some time now. But tracing (and the exploration of that data) is useful enough, which is why it was time to roll it out to other languages & compute offerings.

Customers tracing applications on VMs or containers (including Compute Engine, App Engine flexible environment, and Kubernetes Engine, as well as other non-Google services) can use a new set of instrumentation libraries to submit traces to the service, in a number of languages: Java, Python, NodeJs, Golang, .Net, Ruby, and PHP. We use OpenCensus (a project created by Google) to capture traces from most of these languages, and more languages will gain OpenCensus as time goes on.

What gets traced?

The client-side instrumentation automatically patches well-known modules to insert calls to functions that start, label, and end spans to measure latency of RPCs (such as mysql, redis, etc.) and incoming requests (such as express, hapi, etc.). As each RPC is typically performed on behalf of an incoming request, we must make sure that this association is accurately reflected in span data.

While this happens automatically with the libraries, you can also add custom spans to your code to catch the important bits that fall between.

More info

To get started with Stackdriver APM, simply link the appropriate instrumentation library for each tool to your app and start gathering telemetry for analysis. Stackdriver Debugger and GitHub Enterprise and GitLab, adding to our existing code mirroring functionality for GitHub, Bitbucket, Google Cloud Repositories, as well as locally-stored source code

Understanding the performance of production systems is notoriously difficult. Not only is it difficult to replicate the exact scenario in which the product is being used in the wild, but adding profiling adds an additional load on the production system, which could inadvertently hurt your performance.

This is a catch-22, since if you don’t analyze code execution in production, unexpectedly resource-intensive functions increase the latency and cost of web services every day, without anyone knowing or being able to do anything about it.

This predicament is where many of us find our mental energy : We need to constantly perf-test in prod, but doing so hurts our overall performance.

The trick to solving this is finding a system which can do profiling across all of your deployments while keeping the overhead low.

This is where Stackdriver Profiler comes in.

Stackdriver Profiler

Stackdriver Profiler is a statistical, low-overhead, sampling-based profiler that continuously gathers CPU usage and memory allocation information from your production application instances.

It attributes that information to the source code that generated it, helping you identify the parts of the code that are consuming the most resources, and otherwise illuminating the performance characteristics of the code.

Plus its got a few other features:

1. It analyzes code execution across all environments.
2. It runs continually and uses statistical methods to minimize impact on targeted codebases.
3. It makes it more cost-effective to identify and re-mediate your performance problems rather than scaling up and increasing your monthly bill.

Let’s take a look at it a little closer.

Using it.

Just to show a few things off, I’ve deployed a small Go application in Google Cloud Shell, which imports the `cloud.google.com/go/profiler` package and starts the profiler at the beginning of the application:

Then we just let the program run. Everything else is being taken care of in the background.

Next, in the Google Cloud Platform Console dashboard, go to Profiler:

Which will bring you to the profiler interface.

It then displays this data on a flame chart, presenting the selected metric (CPU time, wall time, RAM used, contention, etc.) for each function on the horizontal axis, with the function call hierarchy on the vertical axis.

You can do cool stuff like CPU, Heap, mutex usage, thread data and other fun things. The UI lets you combine and scrub this data over a nifty time range.

What’s the overhead?

Stackdriver Profiler collects profiling data for 10 seconds every 1 minute for a single instance of a configured service in a single Google Compute Engine zone. For example, if your Kubernetes Engine service runs 10 replicas of a pod If, for example, your Kubernetes Engine service runs 10 replicas of a pod, then each pod will be profiled approximately once in 10 minutes.

The overhead of the CPU and heap allocation profiling at the time of the data collection is less than 5 percent. Amortized over the execution time and across multiple replicas of a service, the overhead is commonly less than 0.5 percent, making it an affordable option for always-on profiling in production systems.

More info

To get started with Stackdriver APM, simply link the appropriate instrumentation library for each tool to your app and start gathering telemetry for analysis. Stackdriver Debugger and GitHub Enterprise and GitLab, adding to our existing code mirroring functionality for GitHub, Bitbucket, Google Cloud Repositories, as well as locally-stored source code

Nomad Socialite was a new geo-location game, where players traveled all around the world to rack up points, catch the latest in urban events, and meet a whole community at the same time. So, catching a plan to San Fran, hitting the hot new Poké popup, and catching their rare collectible digital item can help rack up some huge points.

The challenge, however, was that from a producer perspective, it was getting hard to track analytics on all these events to hand back to the proprietors running events. Once the event size got large enough, the analytics system, (which was powered by Datastore) started to slow significantly when updating.

When you’re talking about Datastore, this type of problem comes up all the time, and is simple to identify: Contention

Review, entity contention

Simply put, Datastore contention occurs when a single entity or entity group is updated too rapidly. The datastore will queue concurrent requests to wait their turn, but if the number of edits on an entity is too high, things will eventually slow down (as the requests stack up) until time-out errors are thrown.

The most common way this limitation gets encountered is when you update an entity with every request — for example, counting the number of views to a page on your site.

Or in the case of Nomad Socialite: Updating the analytics data for an event meant keeping a running count of the number of users logged in, and all their related data sets.

Now, when this happens, there’s lots of ways to address it, from a code perspective (I’ll leave you to this document for some of the more common ones).

From my perspective though, there’s only one good solution : Sharding.

Sharding to the rescue

As we’ve discussed before, Datastore runs on top of Bigtable, and thus is at the mercy of it’s performance characteristics.

If you need to update an entity at a higher rate than Bigtable permits, then you can use replication to accomplish your task.

Sharding effectively duplicates an entity and uses some manual load balancing to write updates to the clones randomly instead of all to the same item. The result is that each clone is partially updated (and the data is not the same between them). When you read, later, fetch all the copies and sum them together on the client. Using this strategy, you would store N copies of the same entity allowing N times higher rate of writes than supported by a single entity.

To prove this, we wrote a small test and then did a 60 minute Load test via Artillery in order to simulate users. In the Google Cloud console, we can see the execution time looks like for trying to update a single entity counter for users.

We can also see that in our sharded version, performance is drastically better:

Notice that the execution time drops considerably since there’s less contention on the entities themselves.

Urban IoT was is a sensor and spatial representation company. Their work is simple, they deploy a whole load of sensors in an urban area that monitor everything from noise, temperature, humidity, pressure, (and more) .They were noticing issues when pushing the time stamped data to GCP Datastore, saying that after a while, their reads slowed down significantly on their dashboards.

Stamped out.

Cloud Datastore is built on top of Google’s NoSQL database, Bigtable, and is subject to Bigtable’s performance characteristics. Bigtable scales by sharding rows into separate tablets (aka a contiguous unit of storage); When a Bigtable tablet experiences a high write rate, the tablet will have to “split” into more than one tablet which is the core component that allows Bigtable to properly scale.

The important point here is this : Sharded rows on separate tablets are lexicographically ordered by key.

By default, Cloud Datastore allocates keys using a scattered algorithm. As such, when values are randomly or even semi-randomly distributed, tablet splits function well. This is because the work to write multiple values is distributed amongst several Bigtable tablets.

However when the keys are lexicographically close, (like a monotonically increasing timestamp…) this work gets isolated into a small update window, and slows things down.

Which, low and behold, was exactly what Urban IoT was doing.

Stamped out.

Remember that for indexed values, we must write corresponding index rows sharded across tablets. If you’re creating new entities at a high rate with monotonically increasing indexed property (like a timestamp), then you’ll run into hotspotting for reads. The result is that the “eventual consistency” model of datastore will take longer to get “consistent” and as performance worsens, read queries will get longer, and eventually cause timeouts.

The take-away about this problem is that hitting this condition is really rare.

To prove this, check out the graph below; It’s a 250QPS load test to datastore, creating entities with keys that are timestamps over a 20 minute window.

It’s not until we jack up the QPS to 750, and run the test for at least an hour, that things start to go south:

Sadly though, this is _exactly_ the situation that Urban IoT was in. They had hundreds of thousands of sensors distributed in an urban environment, all creating new timestamped data at an alarmingly high rate.

The fix is in.

Recall that by default, Cloud Datastore allocates keys using a scattered algorithm Thus you will not normally encounter hotspotting on Cloud Datastore writes if you create new entities at a high write rate using the default ID allocation policy. Typically you only see this type of problem if you’re creating the IDs yourself, and making them linear.

If you do have a key or indexed property that will be monotonically increasing then you can prepend a random hash to ensure that the keys are sharded onto multiple tablets. This is problematic if you plan on doing queries, as you will need to prefix and unprefix the values, then join the results in memory — but it will reduce the error rate of your writes

Doing so let us get performance back down to what they would expect @ the 250QPS rate, without those hot spotting issues.

Last note

And one more tip:

Don’t prematurely optimize for this case, since chances are, you won’t run into it.

It took a couple days of really hammering on Datastore to get our tests to behave the way that Urban IoT was seeing in the wild. So unless you’re in that ballpark, I suggest spending your time in other areas ;)