<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jaana Dogan on Medium]]></title>
        <description><![CDATA[Stories by Jaana Dogan on Medium]]></description>
        <link>https://medium.com/@rakyll?source=rss-1737b4e67578------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6frWC0zC0sYDn68Jfizc9w.jpeg</url>
            <title>Stories by Jaana Dogan on Medium</title>
            <link>https://medium.com/@rakyll?source=rss-1737b4e67578------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:40:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rakyll/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[How to peacefully grow your service]]></title>
            <link>https://rakyll.medium.com/how-to-peacefully-grow-your-service-c40c25fe097f?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/c40c25fe097f</guid>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Wed, 24 Aug 2022 06:59:13 GMT</pubDate>
            <atom:updated>2022-08-24T18:10:20.624Z</atom:updated>
            <content:encoded><![CDATA[<p>In the last almost two decades, I have seen numerous Internet services ranging from small services for niche markets to Tier 1 services for major Internet companies. In this period, apart from database or application design, the technical success of a growing product benefited significantly from adopting a few characteristics. In this article, I’ll capture a few to highlight what helped us.</p><h4>Feature boundaries</h4><p>Small products often start with a single team, single code base, and a single database/storage layer. In the early days, this is essential for the team to make quick adjustments and pivots. Most of the time, early stage services are one or few processes. This allows quick iteration. At some point, your product grows to hundreds of active developers. At this moment, teams may begin to block each other because they have different development velocities, an ever-growing number of building and testing steps, different requirements on how frequently they want to push to production, different levels of prod push success rate, and different levels of availability promises. It would be a failure to adjust the rate of entire development to the slowest team. Instead, teams should be independently building and being able to release without the fear or breaking or blocking others. This is where feature boundaries come into play. Feature boundaries can be designed by either API boundaries or domain layers within boundaries of the same process. Both approaches defines a contract between teams and encapsulate the implementation details. As soon as your API or domain layer keep serving as a contract, you have a path to revise the implementation without any major disruptions to other teams. Achieving continuous delivery without process boundaries/microservices is hard, that’s why most companies end up breaking down their features into new services to be able to operate their lifecycle separately. Feature boundaries are hard to plan for in the early stages, don’t overdo them.</p><h4>Data autonomy</h4><p>Feature boundaries are an essential step to achieve another important milestone, data autonomy. As features grow, teams will likely to recognize that their early choices were not optimal. They will discover new access patterns or customer behavior, or will quickly recognize the initial decisions were not economical from an operability or scalability perspective. At this point, teams would immensely benefit from being able to redesign their databases or storage layer. When data operations are encapsulate behind an API or a domain layer, teams occasionally have a path to make changes without breaking changes or significant regressions. Significant changes may require complete rewrites and may include switching from one database to the other. These changes will require deprecating existing services or domain layers, and migrate users by performing an online migration. API boundaries and domain layers help identifying the extend of your feature set. Strong contracts are a way to document your capabilities, and will assist you when you need to revisit your implementation.</p><p>Data autonomy means the consumers of your data shouldn’t have direct database access to your database, shouldn’t be able to run JOINs between their tables and yours, shouldn’t be able to introduce new data operation without your approval, and doesn’t need to know about the details of your storage layer to be a consumer. It’s almost impossible to take away these permissive characteristics once the ship has sailed.</p><h4>Cell-based architecture</h4><p>A major factor why large Internet companies scale is the cell-based architecture. Cell-based architecture allow to horizontally scale services with reasonable blast radius characteristics. Cell-based architecture is a way to structure your workers and other resources in an often strongly isolated cell that is provisioned by automation. Cells can have different shapes but most commonly they contain a horizontal slice of the product. A cell can serve its own data partition or rely on a regional/global database. Cells with data partitions can fulfill the horizontal sharding needs for databases. Cells that map to data partitions also give teams discipline not to introduce reads/writes that cross partition boundaries. These systems are easy to scale as soon as you have capacity to provision new cells and your external dependencies can handle your new load.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3zNCvEzUxhf-ntWlVMJQjw.png" /><figcaption>A multi-regional cell-based architecture where two services and their data partition is a cell. A user request will be routed to the region and cell the user data lives in.</figcaption></figure><p>Cell-based architecture makes you invest in automation which makes it easier to build new cells and regions as you grow. Cell-based architectures can have tooling in place to migrate tenants among cells in case of a capacity concern.</p><p>Cell-based architectures can be used to provide redundancy and auto failover. A cell could be a replica of another cell, and the load balancer can fallback to the new primary if a cell goes away. Replicas are often run in different availability zones to reduce the impact of a zonal outage.</p><p>Cell-based architectures limit the blast radius in an outage or a security event. Combined with automated continuous delivery practices, it can quickly rollback changes without impacting all of your customers. See <a href="https://aws.amazon.com/builders-library/automating-safe-hands-off-deployments/">Automating safe, hands-off deployments</a> for an example how Amazon uses cells to reduce the impact of bad pushes.</p><h4>Service and database dependencies</h4><p>In the early days of a company, it’s common not to pay attention to the growing list of service dependencies or database calls in the path of a user request. Feature set is small, few service dependencies exist, few data operations are available, and pages are not serving results of tens of database reads. At this point, companies often fail to recognize that the availability of their product is directly related to the availability characteristics of their dependencies. You’ll compromise availability as you are depending on new external dependencies and you cannot beat the availability of your dependencies if they are in your critical path. If an external dependency is available at 99.9% of the time and is in your critical path, the availability of your path is going to be lower. This is why downstream services like databases has to have more aggressive availability targets than your upstream data services or front end servers. <a href="https://queue.acm.org/detail.cfm?id=3096459">The Calculus of Service Availability</a> from Google SRE captures this topic more in depth.</p><p>If you need to rely on a dependency with a lower SLO while maintaining a higher SLO, you can move your calls outside of critical paths (e.g. running them in background jobs), fallback to a default/cheaper behavior, or gracefully degrade the experience.</p><h4>Graceful degradation</h4><p>Graceful degradation is one of the undervalued tools in the early stage of a service when transient errors still have a minor impact. Graceful degradation allow serving a page or even an API call by degrading the experience instead of failing it. Imagine a web page getting bloated with new features in its navigation bar and sidebars in years… Eventually, you may end up finding yourself at a point that it becomes too expensive to serve the page. Minimizing the number of external dependencies would be the best next steps but, you can try to degrade the experience as a short term remedy such as displaying an error message or hiding a section. Graceful degradation may confuse customers if not done correctly, choose how you do it wisely!</p><h4>Unique identifiers</h4><p>If I had a time machine, I’d go back in time and switch <em>some</em> of my identifiers to client-generated globally unique identifiers, GUIDs. The convenience of autoincremented identifiers are great, also their human readable characteristics… But once you grow outside of a single primary database, autoincremented IDs become show stoppers. If you adopt the cell-based architecture with self contained database partitions, it becomes hard to move tenants around. If you’ve exposed the identifiers as links to customers, it becomes impossible to break them. If you’ve represented them as integers in your statistically typed languages or Protobuf files, it becomes a pain to revisit them. The pain of having to do this switch retrospectively is so painful that I often wished we chose GUIDs from the early days for select identifiers.</p><h4>Idempotency</h4><p>When making RPC calls, transient failures such as bad networking may cause calls to be dropped or timed out. RPC calls might be retried and might be received more than once at the server. Idempotency make it safe for the receiver to handle duplicates of request without compromising the integrity of data. Designing idempotent systems are hard, but retrofitting existing systems to become more idempotent is even harder. Idempotency applies to eventing as much as it applies to RPCs. Designing events to be idempotent is a major early contribution.</p><h4>SLOs and quotas</h4><p>Services will have objectives to ensure you don’t degrade the customer experience. Defining them early in your design phase help you to test your design against a theoretical target. It also helps you to eliminate unnecessary dependencies, or dependencies with poor SLOs early in the design phase. SLOs help you to explain the limitations others should consider when they are relying on your service directly. SLOs may help you to evaluate your design at the prototyping phase.</p><p>Quotas or rate limits allow you to reject aggressive call patterns from upstream services. Introducing quotas at a later time, if they are not permissive, could be a breaking change. It’s fairly useful to define them early and set them in place.</p><h4>Observability</h4><p>There are two critical organizationally hard problems in observability: dimensions to collect and propagation standards.</p><p>Dimensions are arbitrary labels collected with telemetry. It allows engineers to narrow down telemetry to identify the blast radius during an outage. For example, being able to break down a service’s latency metrics by Kubernetes cluster name can allow you to see which clusters are affected. Dimensions can be used to see other relevant telemetry. Maybe, you’d like to view logs available for the affected clusters to troubleshoot. Dimensions, when consistently and automatically applied, will improve your incident response and troubleshooting capabilities. Dimensions can get out of the hand if their cardinality becomes too high for metrics. It’s critical to build fundamentals to be able to adjust dimensions.</p><p>Propagation standards, such as a trace context or how dimensions are propagated, are another topic where retrofits are costly and sometimes impossible. The earlier you align in terms of request headers and runtime context propagation, the lesser of a problem it will be.</p><p>—</p><p>The success of Internet services rely on a wide range criteria such as good database design, capacity planning, CI/CD capabilities, understanding the limitations of your service dependencies, and most importantly technical and product talent. In this article, I tried to capture some characteristics that are often undermined but will easily pay off. Some early stage decisions can save years of investments in the long run, and you can use those years building great products for your customers instead of fighting fires.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c40c25fe097f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three Ways to Trace End-to-end]]></title>
            <link>https://rakyll.medium.com/three-ways-to-trace-end-to-end-f61181d6db63?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/f61181d6db63</guid>
            <category><![CDATA[distributed-tracing]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Wed, 07 Jul 2021 20:25:21 GMT</pubDate>
            <atom:updated>2021-07-09T18:14:38.549Z</atom:updated>
            <content:encoded><![CDATA[<p>Having end-to-end distributed traces is a huge challenge for any project. In distributed tracing, end-to-end tracing is a term often used to refer to traces that capture most components in a critical path. Imagine an HTTP request made to trigger a Lambda function. Being able to see the HTTP client, load balancing and scheduling decisions, any outgoing requests made from the Lambda function to serve the request would be “more end-to-end” than just capturing a trace with a function invocation. End-to-end traces are critical because they tell you the larger picture story and how your systems interact with others. They make it easier to diagnose issues, highlight service dependencies, inefficient scheduling or execution patterns, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tmwCr0BmRLIM7vWNMCOYow.png" /><figcaption>On the left, we see a trace only with client and server side spans for a Lambda function trigger. On the right, we see more spans including ALB, and outgoing requests to S3 and to a Redis server.</figcaption></figure><p>There are two main prerequisites to have end-to-end traces:</p><ul><li>Being able to accept and/or propagate the distributed tracing context</li><li>Participating into the incoming trace and producing spans</li></ul><p>The challenge exists because not every component is instrumented to produce distributed traces, and not every component understands distributed tracing context/headers (e.g. <a href="https://github.com/openzipkin/b3-propagation">B3</a>, <a href="https://www.w3.org/TR/trace-context/">W3C TraceContext</a>, X-Amzn-Trace-Id). Even if they do, they may be using different header standards or produce vendor-specific trace data that is hard or impossible to merge. Especially with legacy systems where you can’t modify the source code, or with components without any established context propagation and middleware support, adding tracing or modifying the existing work can be a tremendously expensive investment.</p><h4>Transforming the trace context</h4><p>One of the common difficulties in distributed tracing is the lack of a “standard” propagation header. The community has established options like <a href="https://github.com/openzipkin/b3-propagation">B3</a>, <a href="https://www.w3.org/TR/trace-context/">W3C TraceContext</a> and various vendor-specific headers like X-Amzn-Trace-Id. Projects that are already instrumented to parse one of these formats, is completely unaware of the others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cvRe-YupSIXfgoL-M_IF5g.png" /></figure><p>If service A only understands W3C TraceContext, and service B only understands B3 TraceContext, service B will drop the incoming trace. One intermediate option would be converting the W3C TraceContext to B3 before service B receives the incoming request. If service B is not a the ultimate downstream service or there are other services that require a non-B3 header, you may need to convert the headers back and forth.</p><p><strong>Pros</strong></p><ul><li>It is simple to implement if downstream services consistently are supporting a different header.</li><li>If the converter is implemented as a proxy server, it doesn’t require any changes to the existing services.</li><li>When combined with a vendor-agnostic collection pipeline like the <a href="https://github.com/open-telemetry/opentelemetry-collector">OpenTelemetry collector</a>, services can keep publishing trace spans in vendor specific formats, and the collector can transform and send them to the same service. For example, if service A is using Jaeger and B is using Zipkin, OpenTelemetry collector can both accept spans coming from the services and transform all the data to be sent to Jaeger or Zipkin.</li></ul><p><strong>Cons</strong></p><ul><li>If trace headers are not convertible due to reasons like different identifier lengths or different fields, this option is not possible to implement.</li><li>If downstream services support a variety of different headers, this option is very complicated to implement and maintain.</li></ul><h4>Linking traces</h4><p>In distributed tracing, linking is a concept to be able to associate two or more traces. Even though not all the data is captured under the same trace, when you link traces, you can still navigate from one to the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SOv_wTeBLLwfdDZ7CPFTOw.png" /><figcaption>A trace collected for serviceA.Lookup makes a request to serviceB.Query. The client span links to another trace that contains the serviceB.Query’s server-side traces.</figcaption></figure><p>Assume serviceA and serviceB are instrumented differently and don’t export their tracing data to the same tracing system. In this case, they will have split traces and we can manually link these two traces to make it easier to navigate between them. If serviceB.Query returns its trace ID in the response, the client span of serviceB.Query can record it as a span annotation. Then when browsing the serviceA.Lookup trace, it would be possible to navigate to the linked serviceB.Query trace.</p><p><strong>Pros</strong></p><ul><li>It doesn’t require any significant changes to the existing instrumentation and propagation formats.</li><li>Allows different data access levels. For example, sensitive traces can be split as a linked trace with different access levels.</li></ul><p><strong>Cons</strong></p><ul><li>Requires custom ways to propagate back the “linked trace” and record it.</li><li>Not many distributed tools support visualizing or querying links. Even when querying supported, querying is very limited in comparison to having all the spans under the same trace.</li><li>Hard to produce automatic service maps based on trace data, you may need to build custom solutions.</li><li>Propagating the downsampling decision is still a challenge unless all components are using the same trace header.</li></ul><h4>Partial traces</h4><p>One of the reasonable options would be giving up on “end to end” and only focus on collecting partial traces from the services your group owns and maintains. Even though this is technically not a way to achieve “end-to-end”, it may help the organizations to understand the benefits of having distributed tracing. Even without end-to-end traces, it’s still useful see traces from a specific component to debug and identify issues. This provides a bottom-up approach where a team can communicate the value of distributed traces to the larger organization without having to convenience the entire organization to invest in it.</p><p>—</p><p>Even though, in an ideal world, we would like to have as many as end-to-end traces, it’s not always easy or feasible to have them. These workarounds may be temporary patches but can help you to get the most of distributed tracing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f61181d6db63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why is metric collection still a hard problem in 2020?]]></title>
            <link>https://rakyll.medium.com/why-is-metric-collection-still-a-hard-problem-in-2020-a901ba1e028f?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/a901ba1e028f</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Sun, 06 Dec 2020 18:11:36 GMT</pubDate>
            <atom:updated>2020-12-06T18:16:11.271Z</atom:updated>
            <content:encoded><![CDATA[<p>Metric collection keeps being one of the hard problems. We’ve been collecting metrics for a very long time, so why is this a hard problem still in 2020? Our workloads are becoming larger and more sophisticated. In order to produce useful metric data, we are producing and collecting richer metric data. We are also more interested to understand what else was in the context when the metric was collected. Was there a distributed trace or a log we can correlate? It’s important for us to effectively navigate among different telemetry signals during outages. We have a larger number of metadata items in our modern platforms. Image a container running on Kubernetes… You can associate the container with its cluster, node, pod, service, deployment and more. Additional metadata helps us to identify the origin of the problems, hence we collect metrics with a larger number of labels nowadays.</p><p>I daily find myself explaining why we are still investing a lot of time and energy into metric collection. In this article, I’ll overview some of the most challenging topics that make metric collection still a hard problem.</p><h4>High cardinality labels</h4><p>In the last decade, high cardinality labels became a hot topic in metric collection. High cardinality labels allow you to break down metrics, and allow you to filter and/or aggregate with labels. Without this capability, you have to know a lot more about your system and how it behaves to monitor it. This capability doesn’t only allow you to break down your data in various different dimensions, it allows you to create new dimensions based on existing ones dynamically.</p><p>For example, if you are interested in retrieving metrics only sourced from one service, you can filter your time series and compare them with metrics sourced at other services at your monitoring dashboard. Without high cardinality, you would collect metrics separately in order to achieve the same behavior. In an ever complicated stack where you don’t have access to the source code or cannot redeploy your code each time you need new filters and aggregation, labelling can be critical.</p><p>Unfortunately, there are limits to how much cardinality your collection system can handle. Previously, most metric collection systems and time series storage systems were not built to store and query high cardinality data at all. Most systems are not historically instrumented to ingest high cardinality labels either. Even though high cardinality labels are becoming a bigger necessity, the industry still is struggling with best practices on labelling, we don’t always have capabilities to generate metrics with labels, and we don’t have a metric collection backend to store and query high cardinality metrics. <a href="https://prometheus.io/">Prometheus</a> and <a href="http://www.vldb.org/pvldb/vol13/p3181-adams.pdf">Monarch</a> are two examples of metric collection systems that support labelled metrics.</p><h4>Cross-stack labels</h4><p>Even though metric labelling is seeing more adoption, being able to propagate labels on wire is still an unsolved puzzle. There are no well established propagation standards on wire or in language runtimes.</p><p>Cross-stack labelling is a key technique that differentiates metric collection in microservices environment from the instrumentation of monolithic systems. When microservices systems began to become larger, some practices from the monolithic systems didn’t scale.</p><p>In microservices architectures, it is likely that some services will be a common dependency for a lot of teams. Some examples of these services are authentication and databases that everyone needs and ends up depending on. The solution to this problem is to produce the labels at the upper-level services that calls into the lower-level services. After producing these labels, we will propagate them on the wire as a part of our requests. The lower level services can use the incoming labels when record the metrics. This is how we have fine-grained dimensions at the lower ends of the stack regardless of how many layers of services there are above.</p><p>Without strong propagation standards and first class support from the entire stack, this is a hard problem.</p><h4>Correlation and exemplars</h4><p>In a typical telemetry data collection pipeline, there is often more than just metrics. We instrument our services with a variety of different tools. One example everyone can relate to is logs. Others could be events, distributed traces, runtime profiles coming from production and any other telemetry data you can name.</p><p>Given the large number of variety and the large amount of data, it’s not easy to navigate the collected telemetry data especially during incidents. Being able to see correlated telemetry data from metric dashboards is a great capability.</p><p>Historically, correlation between metrics and other telemetry data didn’t exist in common tools. Our instrumentation libraries didn’t have capabilities to collect exemplars as it’s recording new metric values. OpenCensus is an example instrumentation library that collects trace exemplars with metrics. <a href="https://www.honeycomb.io/blog/get-deeper-insights-with-honeycomb-tracing/">Honeycomb</a> and <a href="https://grafana.com/">Grafana</a> are examples that allow you to navigate to traces from metric dashboards.</p><h4>Export formats</h4><p>Collecting metrics has always been a difficult topic because there are way many ways how services and platforms export metric data. Each year, there are a few new initiatives that is trying to solve this outstanding problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*9nMBMt-OugnruBr_M-WuEQ.png" /></figure><p>As a result, we have many collection paths, and are asking our users to deploy and maintain more than one collection pipeline.</p><h4>Pull vs push</h4><p>In a pull model, your metric collection system is pulling metrics from your services whereas in a push-model your services are pushing the metrics to a metric collection service. This topic is a routine debate topic even though there is no single answer to the problem whether pulling or pushing metrics is a better approach.</p><p>Even though both has its pros and cons, having to support both pull and push in the same collection pipeline is not always an easy task. Prometheus is an example of a pull system whereas Monarch supports pushes and it instructs clients how often they should push. We often have to care about both of the models because our users often need a combination of both.</p><h4>Aggregation</h4><p>Metric collection pipelines often aggregates data because reporting each individual metric collection wouldn’t scale in large systems. Aggregation often happens in multiple layers. Metrics collected from the services can be continuously aggregated along the way until it is in long-term storage. Building and operating aggregation pipelines are not always trivial. Additional to pipelines, there are instrumentation libraries that start aggregating inside the application processes. Figuring out the right aggregation window and fine-tuning aggregations for performance can be a hard problem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a901ba1e028f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What did I forget by working for the same company?]]></title>
            <link>https://rakyll.medium.com/what-did-i-forget-by-working-for-the-same-company-c51b52ce2f8c?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/c51b52ce2f8c</guid>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Sun, 22 Nov 2020 04:23:04 GMT</pubDate>
            <atom:updated>2021-10-23T04:58:45.044Z</atom:updated>
            <content:encoded><![CDATA[<p>My Saturday morning coffee is disrupted by this tweet:</p><p>“xooglers always be like ‘at google we…’ — <a href="https://twitter.com/devonbl/status/1329905383175929858">devonbl</a></p><p>A month into my departure from Google, I can relate to this. It annoys me probably more than my current coworkers at <a href="https://twitter.com/rakyll/status/1315068454873907200">AWS</a> because they know it’s ok to make comparisons when you are a new employee. I avoid all possible conflict of interests. Then, I’m in meetings with open source contributors who are ex-Googlers and so many sentences start with “At Google, we did…” and I add a few more personal stories.</p><p>Let’s be fair. I am candid about portraying the good, the bad and the worst from past experiences. I don’t always enjoy naming companies when referring to experiences, but that’s how you provide context and be honest to your audience.</p><p>I was notoriously known at Google doing this exact thing. “At [insert company], we did…” and that’s how I got my credibility and negotiated what we should do to serve the best for our customers. Believe me, I haven’t seen a dozen companies, but compared to engineers who had Google as their first and only company, I always believed I was bringing more insights. Each company operates widely different, but if you are a platform company or provide world-scale developer products, you care about the experiences of others and the context matters. I joined Google to work there for two years. I wasn’t expecting to stay for eight years and two months. I felt that my time was up a long time ago, but leaving my comfort zone was hard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3QizCD-NKfx-qpE_zIe9XQ.jpeg" /><figcaption>In November 2018, when I was visiting our offices in Europe, I was saying my last goodbyes because I knew that I was probably not coming back.</figcaption></figure><p>My time was up for one exact reason. I no longer had no clue what the life outside Google felt like. My actual superpower was gone. I remember sitting in meetings only bringing insights from what I hear from customers without truly understanding how things worked outside our bubble end-to-end. I didn’t know about the compromises that goes into the decisions or organizational problems other than anecdotes. I didn’t even remember how fast the execution was outside of Google. It felt like I’m at a dead end. If your entire specialization is understanding core problems of the industry and building products to solve them, the lack of exposure to different experiences makes you disrespect yourself. I was waking up in the middle of the night with an overwhelming feel of anxiety, asking what I’m missing by staying. It was late 2018, I decided that something has to change. It was on my way to our Swiss office (my first Google office) and prepared myself saying my last goodbyes to some of the best people I’ve worked in life and telling them I might never be coming back to visit them as I was watching them being sad and sour.</p><p>Then, I left. I could have stayed and watched myself being anxious about what else I’m missing out until I retire. When I left, I had the opportunity to work on anything on any role. I deeply appreciate this privilege, and I’m still not sure if I deserve this kind of recognition... But instead of sitting in my sweet corner, I chose to do more. I decided to put myself in an unknown adventure rather than settling. Once, a friend said, my desire to jump into the unknowns and challenge myself uncomfortably is what makes me differ from everyone he knows.</p><p>I’ve been asked why I left so many times. I can name a couple more reasons, but this is why I left. Change is great.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c51b52ce2f8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Correlation in Latency Analysis]]></title>
            <link>https://rakyll.medium.com/correlation-in-latency-analysis-419357b93287?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/419357b93287</guid>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Thu, 22 Oct 2020 03:14:27 GMT</pubDate>
            <atom:updated>2020-10-22T03:14:27.950Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article was my response to Amazon’s writing assessment when I was interviewed. I answered the question of “What is the most inventive or innovative thing you’ve done? It doesn’t have to be something that’s patented. It could be a process change, product idea, a new metric or customer facing interface — something that was your idea… [retracted]</em></p><p>At Google, our systems are complex and large. Our products rely on many microservices, caches, databases, storage systems and networking infrastructure. In the critical path, a user request may touch up to 100+ components. These components have different deployment and maintenance cycles. Most of our services are built and maintained by different teams. Teams rarely have full insight into each other. Measuring latency, identifying latency issues and being able to debug them effectively is an important capability for the reliability of our systems. Producing actionable telemetry data and improving its navigability have been a critical task for my team. I helped to invent new approaches to <strong>correlate</strong> telemetry signals (exemplars, correlation between tracing and logging, profiler labels) that helped our engineers to navigate latency problems faster.</p><h3>Challenges</h3><p>We define availability as <a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/">Service Level Objectives</a> (SLOs). Latency is a critical element in our SLOs because we describe <em>availability</em> based on customer experience. We measure latency by collecting latency metrics from services. Metrics help us identify SLO violations, but they are not enough for debugging. Once a team receives an alert on a latency issue, they are often relying on <em>additional telemetry</em> <em>signals</em> to further narrow down the source of the problem. Beside metrics, we collect distributed traces, event logs and occasionally other runtime profiles. The additional signals are critical for our engineers to gather more context about the problem. Each signal is useful to identify a category of problems. For example, unexpected numbers of retries are visible in distributed traces, a CPU-heavy library call is visible in our continuous profiling dashboards, poor runtime scheduling is visible in runtime event logs.</p><p>Until recently, it was hard to narrow down the problem by filtering additional signals during an incident. I looked at our surveys to see only 1X% of our engineers said they can find relevant distributed traces easily. We collect tens of millions of traces per second globally, looking for a trace that may have clues about a service’s SLO violation is looking for a needle in a haystack. Additionally, we use different dashboards and visualization tools for metrics, traces, logs, and profiles. Our engineers have to learn the filtering capabilities of 4 or 5 tools in order to search for data relevant to incidents. It is a long and error-prone process. We collect metrics for every user request, but other signals (such as traces, logs, profiles, etc) are downsampled or are optional. It’s sometimes a couple of minutes of work just to see if there had been any additional signals collected for a case.</p><h3>Correlation</h3><p>If an SLO violation happens, our engineers get alerts. Upon receiving an alert, they initially land at the metrics dashboards. Our approach has fundamentally changed how engineers navigate among dashboards. We redesigned our metric collection library to collect a reference to the current trace (if there is a trace in the current context) as it’s collecting a latency metric. This allowed our monitoring dashboards to display latency buckets with exemplar traces. It allowed our engineers to see example traces for each latency bucket on the monitoring dashboards and navigate to traces without friction. Secondly, we started to correlate event logs with traces. It became possible to navigate from a trace to logs with a single click. The final improvement was to introduce the correlation between traces and runtime profiles. In production, we sometimes enable runtime execution tracers to debug language runtimes (e.g. unexpected garbage collector pauses). Runtime events are expensive to collect and runtimes produce a lot of events. If this type of collection is needed, we enable collection very briefly and then disable. Even though the collection is brief, collected data can be hundreds of thousands of events. By linking runtime trace events to distributed traces, we made it possible to navigate from distributed traces to runtime events with a single click.</p><p>Our standard context propagation library played a significant role making this work possible. By propagating the trace headers in the context all through the stack, we enabled all the other instrumentation libraries to recognize the current trace and correlate. It was a large effort to land the rest of the work — an effort of 10+ teams. I led various aspects of coordinating and designing instrumentation library changes, contributing to the data format and service APIs, giving feedback to UX improvements, implementing similar capabilities in our open source stack (e.g. OpenCensus) and playing a role where I advocated for correlation widely in the industry with contributions to several open source projects.</p><h3>Results</h3><p>Our approach to correlate instrumentation signals changed the way we instrument and present data in latency analysis. We made it possible for our engineers to navigate among different telemetry signals without losing context. It now takes several seconds to navigate among telemetry signals that once required minutes. These capabilities also helped us to be able to focus and debug certain customer cases more easily.</p><p>Beside improving incident response, the concept of correlation became a fundamental approach. When it comes to instrumentation, one size doesn’t fit all. When product teams need custom telemetry signals, they correlate their custom signals with the canonical ones. Creating awareness of the availability of the custom signals is less effort now. Correlations also improved the time it takes to hand off systems because it takes less time to onboard new members and teach them how to debug our systems end-to-end. Our work has influenced other instrumentation standards and libraries. I worked closely with the industry where they can benefit from similar concepts. Recently, our approach is adopted by the OpenMetrics initiated by Prometheus, OpenTelemetry, and Go’s execution tracer. <a href="https://medium.com/@rakyll/debugging-latency-in-go-1-11-9f97a7910d68">Go 1.11</a> is released with capabilities to correlate runtime events with distributed traces.</p><p>With minimal changes to existing instrumentation libraries and respecting custom needs, our debugging capabilities and time-to-debug are improved. Today, besides being a fundamental tool in debugging, correlations play a role as a training tool and shortening the time to onboard new engineers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=419357b93287" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spanner’s SQL Story]]></title>
            <link>https://medium.com/google-cloud/spanners-sql-story-79bda8bb632d?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/79bda8bb632d</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cloud-spanner]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Wed, 22 Jul 2020 03:41:53 GMT</pubDate>
            <atom:updated>2020-07-22T18:19:35.737Z</atom:updated>
            <content:encoded><![CDATA[<p>Spanner is a distributed database Google initiated a while ago to build a highly available and highly consistent database for its own workloads. Spanner was initially built to be a key/value and was in a completely different shape than it is today and it had different goals. Since the beginning, it had transactional capabilities, external consistency and was able to failover transparently. Over time, Spanner adopted a strongly typed schema and some other relational database features. In the last years, it added SQL support*. Today we are improving both the SQL dialect and the relational database features simultaneously. Sometimes there is confusion whether Spanner supports SQL or not. The short answer is yes. The long answer is this article.</p><h4>Early years</h4><p>Spanner was initially built for internal workloads and no one was able to see Google is going to initiate a cloud business and externalize Spanner. If you don’t know much about Google, our internal stack is often considered as a different universe. At Google, all systems including storage and database services provide their own proprietary APIs and clients. Are you planning to use your favorite ORM library when you join to Google? Unfortunately, it’s not possible. Our infrastructure services provide their own Stubby/gRPC APIs and client libraries. This is a minor disadvantage if you care about API familiarity but it’s a strong differentiator because we can provide more expressive APIs that represent the differentiated features of our infrastructure.</p><p>Differentiated features are often poorly represented in common APIs. One size doesn’t fit all. Common APIs can only target the common features. Distributed databases are already vastly different than traditional relational databases. I’ll give two examples in this article to show how explicit APIs make a difference.</p><p>Distributed systems fail differently and more often. In order to deal with this problem, distributed systems implement retrying mechanisms. Spanner clients transparently retries transactions when we fail to commit them. This allows us not to surface every temporary failure to the user. We transparently retry with the right backoff strategy.</p><p>In the following snippet, you see some Go code starting a new read-write transaction. It takes a function where you can query and manipulate data. When there is an abort or conflict, it retries the function automatically. <a href="https://godoc.org/cloud.google.com/go/spanner#Client.ReadWriteTransaction">ReadWriteTransaction</a> documents this behavior and documents that the function should be safe to retry (e.g. telling the developers don’t hold application state). This allows us to communicate the unique reality of distributed databases to the user. We can also provide capabilities like auto-retries which are harder to implement in traditional ORMs.</p><pre>import &quot;cloud.google.com/go/spanner&quot;</pre><pre>_, err := client.ReadWriteTransaction(ctx, func(ctx context.Context, txn *spanner.ReadWriteTransaction) error {<br>    // User code here.<br>})</pre><p>Another example is the isolation levels. Spanner implements an isolation level better than the strictest isolation level (serializable) described in the SQL standard. Spanner doesn’t allow you to pick anything less strict for read/write transactions. But for multi-regional setups and read-only transactions, providing the strongest isolation is not always feasible. Our limits are tested by the speed of the light. For users who are ok with slightly stale data, Spanner has capabilities to provide stale reads. Stale reads allow users to read the version available in the regional replica. They can set how much staleness they can tolerate. For example, the transaction below can tolerate up to 10 seconds.</p><pre>import &quot;cloud.google.com/go/spanner&quot;</pre><pre>client.ReadOnlyTransaction().<br>    WithTimestampBound(spanner.MaxStaleness(10*time.Second)).<br>    Query(ctx, query)</pre><p>Staleness API allows us to explicitly express how snapshot isolation works and how Spanner can go and fetch the latest data if replica is very out of date. It also allows us to highlight how multi-regional replication is a hard problem and even with a database like Spanner, you may consider compromising from consistency for better latency characteristics in a multi-regional setup.</p><h4>F1</h4><p><a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41344.pdf">F1</a> was the original experiment for the first steps towards having SQL support in Spanner. F1 is a distributed database at Google that is built on top of Spanner. Unlike Spanner, it supported:</p><ul><li>Distributed SQL queries</li><li>Transactionally consistent secondary indexes</li><li>Change history and stream</li></ul><p>It implemented these features in a coordination layer on top of Spanner and handed off everything else to Spanner.</p><p>F1 was built to support our Ads products. Given the nature of the ads business and the complexity of our Ads products, being able to write and run complex queries was critical. F1 made Spanner more accessible to business-logic heavy systems.</p><h4>Spanner in Cloud</h4><p>Fast forward, Google Cloud launched Spanner for our external customers. When it was first released, it only had SQL support to query data. It lacked INSERT, UPDATE and DELETE statements.</p><p>Given it didn’t fully a SQL database back then, it also lacked driver support for <a href="https://cloud.google.com/spanner/docs/jdbc-drivers">JDBC</a>, <a href="https://github.com/rakyll/go-sql-driver-spanner">database/sql</a> and similar. Driver support became a possibility when Cloud Spanner <a href="https://cloud.google.com/spanner/docs/release-notes#October_10_2018">implemented</a> a Data Manipulation Language (DML) support for inserts, updates and deletes.</p><p>Today, Cloud Spanner supports both DDLs (for schema) and DMLs (for data). Cloud Spanner uses a SQL dialect used by Google. <a href="https://github.com/google/zetasql">ZetaSQL</a>, a native parser and analyzer of this dialect has open sourced a while ago. As of today, Cloud Spanner also provides a query analysis tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ysDhwpJtBopIRRF9m4xfw.png" /><figcaption>An example query analysis from Spanner’s query analyzer.</figcaption></figure><h4>Next?</h4><p>There are current challenges originated from the differences present in our SQL dialect. This is an area we are actively trying to improve. Not just we don’t want our users to deal with a new SQL flavor, the current situation is also slowing down our work on ORM integrations. Some of the ORM frameworks hardcode SQL when generating queries and giving drivers little flexibility to override behavior. In order to avoid any inconsistencies, we are trying to close the differences with popular dialects.</p><p>Dialect differences are not the only problem affecting our SQL support. One other significant gap is the lack of some of the common traditional database features. Spanner never supported features like default values or autogenerated IDs. As we are improving the dialect differences, it’s always in our radar to simultaneously address these significant gaps.</p><p>—</p><p>(*) The initial work on Spanner’s querying work is published as <a href="https://research.google/pubs/pub46103/">Spanner: Becoming a SQL System</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79bda8bb632d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/spanners-sql-story-79bda8bb632d">Spanner’s SQL Story</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How does Spanner avoid single point of failures in writes?]]></title>
            <link>https://medium.com/google-cloud/how-does-spanner-avoid-single-point-of-failures-in-writes-4f7765cd894?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f7765cd894</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cloud-spanner]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Tue, 19 May 2020 02:33:45 GMT</pubDate>
            <atom:updated>2020-05-28T01:32:06.065Z</atom:updated>
            <content:encoded><![CDATA[<h3>How Does Spanner Avoid Single Point of Failures in Writes?</h3><p>Google’s <a href="https://cloud.google.com/spanner">Spanner</a> is a relational database with 99.999% availability which translates to 5 mins of downtime a year. Spanner is a distributed system and can span multiple machines, multiple datacenters (and even geographical regions when configured). It splits the records automatically among its replicas and provides automatic failover. Unlike traditional failover models, Spanner doesn’t failover to a secondary cluster but can elect an available read-write replica as the new leader.</p><p>In relational databases, providing both high availability and high consistency in writes is a very hard problem. Spanner’s synchronous replication, the use of dedicated networking and <a href="https://en.wikipedia.org/wiki/Paxos_(computer_science)">Paxos</a> voting provides high availability without compromising consistency.</p><h4>High availability of reads vs writes</h4><p>In traditional relational databases (e.g. MySQL or PostgreSQL), scaling and providing higher availability to reads is easier than writes. Read-only replicas provide a copy of the data read-only transactions can retrieve from. Data is replicated to the read-only replicas from a read-write master either synchronously or asynchronously.</p><p>In synchronous models, master synchronously writes to the read replicas at each write. Even though this model ensures that read-only replicas always have the latest data, it makes the writes quite expensive (and causes availability issues for writes) because the master has to write to all available replicas before it returns.</p><p>In asynchronous models, read-only replicas get the data from a stream or a replication log. Asynchronous models make writes faster but introduce a lag between the master and the read-only replicas. Users have to tolerate the lag and should be monitoring it to identify replication outages. The asynchronous writes make the system inconsistent because not all the replicas will have the latest version of the until asynchronous synchronization is complete. The synchronous writes make data consistent by ensuring all replicas got the change before a write succeeds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*mFqYWVw_E65LorCHbAX-GQ.png" /><figcaption>Synchronous vs asynchronous replication to read-only replicas.</figcaption></figure><p>Horizontally scaling reads by adding more read replicas is only part of the problem. <strong>Scaling writes is a harder problem</strong>. Having more than one master is introducing additional problems. If a master is having outage, other(s) can keep serving writes without users experiencing downtime. This model requires replication of writes among masters. Similar to read replication, <a href="https://en.wikipedia.org/wiki/Multi-master_replication">multi-master replication</a> can be implemented asynchronously or synchronously. If implemented synchronously, it often means less availability of writes because a write should replicate in all masters and they should be all available before it can succeed. As a tradeoff, multi-master replication is often implemented with asynchronous replication but it negatively impacts the overall system by introducing:</p><ul><li>Looser consistency characteristics that violate ACID promises.</li><li>Increased risk of timeouts and communication latency.</li><li>Necessity for conflict resolution between two or more masters if conflicting updates happened but not communicated.</li></ul><p>Due to the complexity and the failure modes multi-master replication introduces, it’s not a commonly preferred way of providing high availability in practice.</p><p>As an alternative, <a href="https://en.wikipedia.org/wiki/High-availability_cluster">high-availability clusters</a> are a more popular choice. In this model, you’d have an entire cluster that can take over when the primary master goes down. Today, cloud providers implement this model to provide high availability features for their managed traditional relational database products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*kppaURSo0vDe_qvf0Ycxmw.png" /><figcaption>An example model where secondary cluster can take over if primary has an outage, load balancer will point to the secondary and all writes and reads will be served from the secondary.</figcaption></figure><h4>Topology</h4><p>Spanner doesn’t use high availability clusters but approaches to the problem from a different angle. A Spanner cluster* <a href="https://cloud.google.com/spanner/docs/replication#replica_types">contains</a> multiple read-write, may contain some read-only and some witness replicas.</p><ul><li>Read-write replicas serve reads and writes.</li><li>Read-only replicas serve reads.</li><li>Witnesses don’t serve data but participate in leader election.</li></ul><p>Read-only and witness replicas are only used for multi-regional Spanner clusters that can span across multiple geographical regions. Single region clusters only use read-write replicas. Each replica lives in a different zone in the region to avoid single point of failure due to zonal outages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*R3NRckjvQBV6qZm3LgXwWQ.png" /><figcaption>The write is routed to the leader**. Then the write is synchronously replicated to other replicas.</figcaption></figure><h4>Splits</h4><p>Spanner’s replication and sharding capabilities come from its splits. Spanner splits data to replicate and distribute them among the replicas. Split happens <a href="https://cloud.google.com/spanner/docs/schema-and-data-model#load-based_splitting">automatically</a> when Spanner detects high read or high write load among the records. Each split is replicated and has a <strong>leader replica</strong>.</p><p>When a write arrives, Spanner finds the split the row is in. Then, we look for the leader of that split and route the write to the leader. This is <em>true</em> even in multi-region setups where user is geographically closer to another non-leader read-write replica. In the case of an outage of the leader, an available read-write replica is elected as the leader and user’s write is served from there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*qimBK_B4NPAhmFkbxUEIDQ.png" /><figcaption>Each split is replicated three times and only one of the replicas is the split leader. Each split may be located in a different machine, see <a href="https://cloud.google.com/spanner/docs/whitepapers/life-of-reads-and-writes#practical_example">https://cloud.google.com/spanner/docs/whitepapers/life-of-reads-and-writes</a>.</figcaption></figure><p>In order for a write to succeed, a leader needs to synchronously replicate the change to the other replicas. But isn’t this impacting the availability of the writes negatively? If writes need to wait for all replicas to succeed, a replica can be a single point of failure because writes wouldn’t succeed until all replicas replicate the change.</p><p>This is where Spanner does something better. Spanner only requires <em>a majority of the Paxos voters to successfully write</em>. This allows writes to succeed even when a read-write replica goes down. Only the majority of the voters are required not all of the read-write replicas.</p><h4>Synchronous replication</h4><p>As mentioned above, synchronous replication is hard and impacts the availability of the writes negatively. On the other hand when replication happens asynchronously, they cause inconsistencies, conflicts and sometimes data loss. For example, when a master becomes unavailable due to a networking issue, it may still have committed changes but might have not delivered them to the secondary master. If the secondary master updates the same records after a failover, data loss can happen or conflict resolution may be required. PostgreSQL provides <a href="https://momjian.us/main/writings/pgsql/replication.pdf">a variety</a> of replication models with different tradeoffs. The tradeoffs summary below can give you a very high level idea of how many different concerns to worry about when designing replication models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vwpl0Tmf2eqGzKQvPNQeOg.png" /><figcaption>A <a href="https://momjian.us/main/writings/pgsql/replication.pdf">summary</a> of various PostgreSQL replication models and their tradeoffs.</figcaption></figure><p>Spanner’s <a href="https://cloud.google.com/spanner/docs/replication">replication</a> is <em>synchronous</em>. Leaders have to synchronously communicate with other read/write replicas about the change and confirm it in order for a write to succeed.</p><h4>Two-phase commit (2PC)</h4><p>While writes only affecting a single split uses a simpler and faster protocol, if two or more splits are required for a write transaction, <a href="https://en.wikipedia.org/wiki/Two-phase_commit_protocol">two-phase commit</a> (2PC) is executed. 2PC is infamously known as “the anti-availability protocol” because it requires participation from all the replicas and any replica can be a single point of failure. Spanner still serves writes even if some of the replicas are unavailable, because only a majority of voting replicas are required in order to commit a write.</p><h4>Network</h4><p>Spanner is a distributed system and is inherently affected by problems that are impacting distributed systems in general. Networking itself is a factor of outages in distributed systems. On the other hand, Google cites only <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf">7.6% </a>of the Spanner failures were networking related. This is mostly because it runs on Google’s private network. Years of operational maturity, reserved resources, having control over upgrades and hardware makes networking not a significant source of outages. <a href="https://cloud.google.com/blog/products/gcp/inside-cloud-spanner-and-the-cap-theorem">Eric Brewer’s earlier article</a> explains the role of networking in this case more in detail.</p><h4>Colossus</h4><p>Spanner’s durability guarantees come from Google’s distributed file system, Colossus. Spanner also mitigates some more risk by depending on Colossus. The use of Colossus allows us to have the file storage decoupled from the database service. Spanner is a “shared nothing” architecture and because any server in a cluster can read from Colossus, replicas can recover quickly from whole-machine failures.</p><p>Colossus also provides replication and encryption. If a Colossus instance goes down, Spanner can still work on the data via the available Colossus instances. Colossus encrypts data and this is why Spanner provides encryption at rest by default out of the box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*RO68d75Iu_rdYvhJkxSx1g.png" /></figure><p>Spanner read-write replicas hands off the data to Colossus where data is replicated for 3 times. Given there are three read-write replicas in a Spanner cluster, this means the data is replicated for 9 times.</p><h4>Automatic retries</h4><p>As repeatedly mentioned above, Spanner is a distributed system and is not magic. It experiences more internal aborts and timeouts than traditional databases when writing. A common strategy in distributed systems in order to deal with partial and temporary failures is to retry. Spanner client libraries <em>provide automatic retries for read/write transactions</em>. In the following Go snippet, you see the APIs to create a read-write transaction. The client automatically retries the body if it fails due to aborts or conflicts:</p><pre>import &quot;cloud.google.com/go/spanner&quot;</pre><pre>_, err := client.ReadWriteTransaction(ctx, func(ctx context.Context, txn *spanner.ReadWriteTransaction) error {<br>    <strong>// User code here.</strong><br>})</pre><p>One of the challenges of developing ORM framework support for Google Cloud Spanner was the fact most ORMs didn’t have automatic retries, therefore their APIs didn’t give developers a sense that they shouldn’t maintain any application state in the scope of a transaction. In contrast, Spanner libraries care a lot of retries and make an effort to automatically deliver them without creating extra burden to the user.</p><p>—</p><p>Spanner approaches to sharding and replication differently than traditional relational databases. It utilizes Google’s infrastructure and fine-tunes several traditionally hard problems to provide high availability without compromising consistency.</p><p><strong>Footnotes:</strong><br>(*) Google Cloud Spanner’s terminology for a cluster is an instance. I avoided to use “instance” because it is an overloaded term and might mean “replica” for the majority of the readers of this article.<br>(**) The write is routed to the <em>split</em> leader. Read the Splits section for more.</p><p>This article is archived at <a href="https://spanner.fyi/ha-writes/">spanner.fyi/ha-writes</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f7765cd894" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/how-does-spanner-avoid-single-point-of-failures-in-writes-4f7765cd894">How does Spanner avoid single point of failures in writes?</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Things I Wished More Developers Knew About Databases]]></title>
            <link>https://rakyll.medium.com/things-i-wished-more-developers-knew-about-databases-2d0178464f78?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d0178464f78</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Tue, 21 Apr 2020 18:02:44 GMT</pubDate>
            <atom:updated>2020-04-23T06:12:12.446Z</atom:updated>
            <content:encoded><![CDATA[<p>A large majority of computer systems have some state and are likely to depend on a storage system. My knowledge on databases accumulated over time, but along the way our design mistakes caused data loss and outages. In data-heavy systems, databases are at the core of system design goals and tradeoffs. Even though it is impossible to ignore how databases work, the problems that application developers foresee and experience will often be just the tip of the iceberg. In this series, I’m sharing a few insights I specifically found useful for developers who are not specialized in this domain.</p><ol><li>You are lucky if 99.999% of the time network is not a problem.</li><li>ACID has many meanings.</li><li>Each database has different consistency and isolation capabilities.</li><li>Optimistic locking is an option when you can’t hold a lock.</li><li>There are anomalies other than dirty reads and data loss.</li><li>My database and I don’t always agree on ordering.</li><li>Application-level sharding can live outside the application.</li><li>AUTOINCREMENT’ing can be harmful.</li><li>Stale data can be useful and lock-free.</li><li>Clock skews happen between any clock sources.</li><li>Latency has many meanings.</li><li>Evaluate performance requirements per transaction.</li><li>Nested transactions can be harmful.</li><li>Transactions shouldn’t maintain application state.</li><li>Query planners can tell a lot about databases.</li><li>Online migrations are complex but possible.</li><li>Significant database growth introduces unpredictability.</li></ol><p><em>Thanks much to Emmanuel Odeke, Rein Henrichs and others for their review and feedback on an earlier version this article.</em></p><h4>You are lucky if 99.999% of the time network is not a problem.</h4><p>It’s an open debate how reliable today’s networking is and how commonly systems experience downtime because of networking outages. The available research is limited and is often dominated by large organizations who have dedicated networking with custom hardware, as well as specialized staff.</p><p>With 99.999% service availability, Google cites only <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf">7.6%</a> of Spanner (Google’s globally distributed database) issues are caused by networking even though it keeps crediting its dedicated networking as a core reason behind its availability. <a href="https://cacm.acm.org/magazines/2014/9/177925-the-network-is-reliable/fulltext">Bailis’ and Kingsbury’s survey</a> from 2014 is challenging one of the <a href="https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing">Fallacies of Distributed Computing</a> coined by Peter Deutsch in 1994. Is network really reliable?</p><p>We don’t have comprehensive survey outside of giants or over the public Internet. There is also not enough data from major providers how much of their customers issues can be traced back to networking problems. We often experience outages in large cloud provider’s networking stack can take parts of the Internet down for hours but these are only the high-impact events where a large number of visible customers are impacted. Networking outages might be affecting more cases even though not all events are making much noise. Cloud customers don’t necessarily have visibility into their problems either. When there is an outage, identifying it as a networking error caused in the provider is not possible. To them, third-party services are black boxes. Estimating the impact without being a major provider is not possible.</p><p>In comparison to what major players report on their systems, it might be safe to say you are lucky if networking issues represents a small percentage of your potential problems that cause outage. Networking still suffer from conventional issues such as hardware failures, topology changes, administrative configuration changes and power failures. But I recently learned that newly discovered problems such as <a href="https://twitter.com/rakyll/status/1249891472993693696">SHARK BITES</a> (yes, shark bites) are a reality.</p><h4>ACID has many meanings.</h4><p>ACID stands for atomicity, consistency, isolation, durability. These are the properties database transactions need to guarantee to their users for validity even in the event of crash, error, hardware failures and similar. Without ACID or similar contracts, application developers wouldn’t have a guidance on what’s their responsibility versus what the databases provide. Most relational transactional databases are trying to be ACID-compliant, but new approaches such as NoSQL movement gave birth to many databases without ACID transactions because they are expensive to implement.</p><p>When I was new in the industry, our tech lead was arguing whether ACID is an obsolete concept or not. It is fair to say ACID is considered a loose description instead of a strict implementation standard. Today, I find it mostly useful because it provides a category of problems (and a category of possible solutions).</p><p>NOT every database is ACID-compliant and among ACID-compliant databases, ACID can be interpreted differently. One of the reasons why ACID is implemented differently is the number of tradeoffs involved in implementing ACID capabilities. Databases might advertise themselves as ACID but might still have different interpretation in edge cases or how they handle “unlikely” events. Developers can at least learn at a high-level how databases implement things in order to have a proper understanding of fail modes and design tradeoffs.</p><p>One well-known debate is how ACID MongoDB is even after v4. MongoDB didn’t have <a href="https://www.mongodb.com/blog/post/what-about-durability">journaling</a> support for a long time even though it was not committing data files to disk not more frequently (every 60 seconds) by default. Consider the following scenario, application makes two writes (w1 and w2). MongoDB was able to persist the change for the first write, but it fails to do it for w2 because it crashes due to a hardware failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*2QPkOUcin02S3zX9BDz3qw.png" /><figcaption>An illustration of data loss if MongoDB crashes before it writes to the physical disk.</figcaption></figure><p>Committing to disk is an expensive process and by avoiding commits, they were claiming to be performant in writes while sacrificing durability. As of today, MongoDB has journaling but dirty writes still can affect the durability of data because they are committing journals at every 100ms by default. The same scenario is still possible for the durability of the journals and changes represented in those logs even though the risk is significantly less.</p><h4>Each database has different consistency and isolation capabilities.</h4><p>Among ACID properties, consistency and isolation have the widest spectrum of different implementation details because the spectrum of tradeoffs is wider. Consistency and isolation are expensive capabilities. They require coordination and are increasing contention in order to keep data consistent. When having to horizontally scale among data centers (especially among different geographic regions), the problems become significantly harder. Providing high levels of consistency can be extremely hard as availability decreases and networking partitions happen more often. See the <a href="https://en.wikipedia.org/wiki/CAP_theorem">CAP theorem</a> for a more general explanation of this phenomena. It is worth to also note that applications can handle a bit of inconsistency or programmers might have enough insights about the problem to add additional logic in the application to handle it without heavily relying on their database.</p><p>Databases often provide a variety of isolation layers so the application developers can pick the most cost effective one based on their tradeoffs. Weaker isolation can be faster but may introduce data races. Stronger isolation eliminates some potential data races but will be slower and might introduce contention that will slow down the database to a point it may cause outages.</p><figure><a href="https://jepsen.io/consistency"><img alt="" src="https://cdn-images-1.medium.com/max/759/1*x95aVFq-wMB6VrI9AD4k8Q.png" /></a><figcaption>An overview of the existing concurrency models and the relationships between them.</figcaption></figure><p>The SQL standard only defines four isolation levels even though there are more levels theoretically and practically available. <a href="https://jepsen.io/consistency">jepson.io</a> provides a compelling overview of the existing concurrency models if you need further reading. For example, Google’s Spanner guarantee external serializability with clock synchronization and even though this is a stricter isolation layer, it is not defined in the standard isolation layers.</p><p>The isolation levels mentioned in the SQL standard are:</p><ul><li><strong>Serializable</strong> (most strict, expensive): A serializable execution produces the same effect as some serial execution of those transactions. A serial execution is one in which each transaction executes to completion before the next transaction begins. One note about <em>Serializable</em> level is that it is often implemented as “snapshot isolation” (e.g. Oracle) due to differences in interpretation and “snapshot isolation” is not represented in the SQL standard.</li><li><strong>Repeatable reads</strong>: Uncommitted reads in the current transaction are visible to the current transaction but changes made by other transactions (such as newly inserted rows) won’t be visible.</li><li><strong>Read committed</strong>: Uncommitted reads are not visible to the transactions. Only committed writes are visible but the phantom reads may happen. If another transaction inserts and commits new rows, the current transaction can see them when querying.</li><li><strong>Read uncommitted</strong> (least strict, cheap): Dirty reads are allowed, transactions can see not-yet-committed changes made by other transactions. In practice, this level could be useful to return approximate aggregates, such as COUNT(*) queries on a table.</li></ul><p>Serializable level allows least opportunities for data races to happen although being the most expensive and introduces the most contention to the system. Other isolation levels are cheaper but increases the possibility of data races. Some databases allow you to set your isolation level, some databases are more opinionated about them and not necessarily supporting all of them.</p><p>Even though databases advertise their support for these isolation levels, a careful examination of their behavior may provide more insights on what the actually do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*UUEYw0PyXpyGFcT9OsDeuA.png" /><figcaption>An overview of concurrency anomalies at different isolation levels per database.</figcaption></figure><p>Martin Kleppmann’s <a href="https://github.com/ept/hermitage">hermitage</a> provides an overview of different concurrency anomalies and whether a database is able to handle it at a particular isolation level. Kleppmann’s research shows how isolation levels can be interpreted differently by database designers.</p><h4>Optimistic locking is an option when you can’t hold a lock.</h4><p>Locks can be extremely expensive not only because they introduce more contention in your database but they might require consistent connections from your application servers to the database. Exclusive locks can be effected by network partitions more significantly and cause deadlocks that are hard to identify and resolve. In cases where being able to hold exclusive locks is not easy, optimistic locking is an option.</p><p><a href="https://convincedcoder.com/2018/09/01/Optimistic-pessimistic-locking-sql/#optimistic-locking-using-where">Optimistic locking</a> is a method when you read a row, you take note of a version number, last modified timestamps or its checksum. Then you can check the version hasn’t changed atomically before you mutate the record.</p><pre>UPDATE products<br>SET name = &#39;Telegraph receiver&#39;, version = 2 <br>WHERE id = 1 AND version = 1</pre><p>Update to products table is going to affect 0 rows if another update has changed this row earlier. If no earlier updates have been done, it will affect 1 row and we can tell our update has succeeded.</p><h4>There are anomalies other than dirty reads and data loss.</h4><p>When we are talking about data consistency, we primarily pay a lot of attention to possible race conditions that can lead to dirty reads and data loss. But anomalies with data are not just limited to them.</p><p>An example of this type of anomalies is write skews. Write skews are harder to identify because we are not actively looking for them. Write skews are caused not when dirty reads happen on writes or lost but logical constraints on data is compromised.</p><p>For example, assume a monitoring application that requires one person among their operators to be oncall all the times.</p><pre>BEGIN tx1;                      BEGIN tx2;</pre><pre>SELECT COUNT(*) <br>FROM operators<br>WHERE oncall = true;<br>0                               SELECT COUNT(*)<br>                                FROM operators<br>                                WHERE oncall = TRUE;<br>                                0</pre><pre>UPDATE operators                UPDATE operators<br>SET oncall = TRUE               SET oncall = TRUE<br>WHERE userId = 4;               WHERE userId = 2;</pre><pre>COMMIT tx1;                     COMMIT tx2;</pre><p>In the situation above, there will be a write skew if two of the transactions successfully commit. Even though no dirty read or data loss happened, the integrity of data is lost because there are two people assigned to be oncall.</p><p>Serializable isolation, schema design or database constraints can be helpful to eliminate write skews. Developers need to be able to identify such anomalies during development to avoid data anomalies in production. Having said that, identifying write skews in code bases can be extremely hard. Especially in large systems, if different teams are responsible for building features based on same tables without talking to each other and examining how they access the data.</p><h4>My database and I don’t always agree on ordering.</h4><p>One of the core capabilities databases offer is the ordering guarantees but ordering may be surprising to the application developer. Databases see transactions in the order they receive them not in the programming order developers see them. The order of the transaction execution is hard to predict especially in high-volume concurrent systems.</p><p>In development time, especially when working with non-blocking libraries, poor style and readability may contribute to the problem where users think transactions are executed sequentially even though they can arrive at the database at any order. The program below makes it look like T1 and T2 are going to be invoked sequentially, but if these functions are non-blocking and return immediately with a promise, the order of the invocation will be up to the time they have received at the database.</p><pre>result1 = T1() // results are actually promises<br>result2 = T2()</pre><p>If atomicity is required (to either fully commit or abort all operations) and the sequence matter, the operations in T1 and T2 should run in a single database transaction.</p><h4>Application-level sharding can live outside the application.</h4><p>Sharding is a way to horizontally partition your database. Even though some databases can automatically partition data horizontally, some don’t or may not be good at it. When data architects/developers can predict how data is going to be accessed, they might create horizontal partitions at the user-land instead of delegating this work to their database. This is called application-level sharding.</p><p>The name, application-level sharding, often gives the wrong impression that sharding should live in the application services. Sharding capabilities can be implemented as a layer in front of your database. Depending on data growth and schema iterations, sharding requirements might get complicated. Being able to iterate on some strategies without having to redeploy application servers may be useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*8_yDPQbGxMb7Zv_UBs4pTQ.png" /><figcaption>An example architecture where application servers are decoupled from the sharding service..</figcaption></figure><p>Having sharding as a separate service can increase your capabilities on iterating on sharding strategies without having to redeploy your applications. One such example of an application-level sharding system is <a href="https://youtu.be/OCS45iy5v1M?t=204">Vitess</a>. Vitess provides horizontal sharding for MySQL and allows clients to connect to it via the MySQL protocol and it shards the data on various MySQL nodes that don’t know about each other.</p><h4>AUTOINCREMENT’ing can be harmful.</h4><p>AUTOINCREMENT’ing is a common way of generating primary keys. It’s not uncommon to see cases where databases are used as ID generators and there are ID-generation designated tables in a database. There are a few reasons why generating primary keys via auto-incrementing may not be not ideal:</p><ul><li>In distributed database systems, auto-incrementing is a hard problem. A global lock would be needed to be able to generate an ID. If you can generate a UUID instead, it would not require any collaboration between database nodes. Auto-incrementing with locks may introduce contention and may significantly downgrade the performance for insertions in distributed situations. Some databases like MySQL may require specific <a href="https://www.percona.com/blog/2011/01/12/conflict-avoidance-with-auto_increment_incremen-and-auto_increment_offset/">configuration</a> and more attention to get things right in master-master replication. The configuration is easy to mess up and can lead to write outages.</li><li>Some databases have partitioning algorithms based on primary keys. Sequential IDs may cause unpredictable hotspots and may overwhelm some partitions while others stay idle.</li><li>The fastest way to access to a row in a database is by its primary key. If you have better ways to identify records, sequential IDs may make the most significant column in tables a meaningless value. Please pick a globally unique natural primary key (e.g. a username) where possible.</li></ul><p>Please consider the impacts of auto-incremented IDs vs UUIDs on indexing, partitioning and sharding before you decide on what works better for you.</p><h4>Stale data can be useful and lock-free.</h4><p>Multi-version concurrency control (MVCC) enables a lot of the consistency features we briefly discussed above. Some databases (e.g. Postgres, Spanner) uses MVCC to allow each transaction to see a snapshot, an older version of the database. Transactions against snapshots still can be serializable for consistency. When reading from an old snapshot, you read stale data.</p><p>Reading slightly stale data would be useful, for example when you are generating analytics from your data or calculating approximate aggregate values.</p><p>The first advantage of reading stale data would be latency (especially if your database is distributed among different geographical regions). The second advantage of a MVCC database is that it would allow read-only transactions to to be lock-free. A major advantage in a read-heavy application if the stale data can be tolerated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*ePJGd32VU4esxofh5LP69Q.png" /><figcaption>Application server reads 5-second old stale data from local replica even though the latest version is available on the other side of the Pacific Ocean.</figcaption></figure><p>Databases sweep the old versions automatically and in some cases, they allow you to do that on demand. For example, Postgres allows users to VACUUM on demand as well as automatically vacuuming once a while, and Spanner runs a garbage collector to get rid of the versions older than an hour.</p><h4>Clock skews happen between any clock sources.</h4><p>The most well-hidden secret in computing is that all time APIs lie. Our machines don’t accurately know what the current time is. Our computers all contain a quartz crystal that produces a signal to tick time. But quartz crystals can’t accurately tick and drift in time, either faster or slower than the actual clock. Drift could be up to 20 seconds a day. The time on our computers need to be synchronized by the actual time every now and then for accuracy.</p><p>NTP servers are used for synchronization but synchronization itself could be delayed due to network. When synchronizing with an NTP server in the same data center can take time, syncing with a public NTP server may cause more skew.</p><p>Atomic and GPS clocks are better sources to determine the current time but they are expensive and need complicated setup that they cannot be installed on every machine. Given the limitations, in data centers, a multi-tiered approach is used. While atomic and/or GPS clocks are providing accurate timing, their time is broadcasted to the rest of the machines via secondary servers. This means every machine will be drifted from the actual current time with some magnitude.</p><p>There is more… Applications and databases often live in different machines (if not in different centers). Not just that database nodes distributed in a few machines won’t be able to agree on what the time is, application server clock and a database node clock won’t agree either.</p><p>Google’s TrueTime is following a different approach here. Most people think Google’s progress in clocks can be attributed to their use of atomic and GPS clocks, but that’s only the part of the story. This is what TrueTime does:</p><ul><li>TrueTime uses two different sources: GPS and atomic clocks. These clocks have different fail modes, hence using both of them is increasing the reliability.</li><li>TrueTime has an unconventional API. It returns the time as an interval. The time could be in fact anywhere between the lower bound and the upper bound. Google’s distributed database Spanner then can wait until it is certain the current time is beyond a particular time. This method adds some latency to the system especially when the uncertainty advertised by masters are high but provides correctness even in a globally distributed situation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*PT3Xi-fBznMiYtH9_EmJUA.png" /><figcaption>Spanner components use TrueTime where TT.now() returns an interval, so Spanner can inject sleeps to ensure the current time has passed a particular timestamp.</figcaption></figure><p>As the confidence on the current time decreases, it means Spanner operations might take more time. This is why even though having accurate clocks would be impossible, it is still important to keep the confidence high for performance.</p><h4>Latency has many meanings.</h4><p>If you ask ten people in a room what “latency” means, they may all have different answers. In databases, latency is often referred to “database latency” but not the latency client perceives. Client will see a latency of database latency and network latency. Being able to identify client and database latency is critical when debugging escalating problems. When collecting and displaying metrics, always consider having both.</p><h4>Evaluate performance requirements per transaction.</h4><p>Sometimes databases advertise their performance characteristics and limitations in terms of write and read throughput and latency. Although this may give a high level overview of the major blockers, when evaluating a new database for performance, a more comprehensive approach is to evaluate critical operations (per query and/or per transaction) separately. Examples:</p><ul><li>Write throughput and latency when inserting a new row in to table X (with 50M rows) with given constraints and populating rows in related tables.</li><li>Latency when querying the friends of friends of a user when average number of friends is 500.</li><li>Latency of retrieving the top 100 records for the user timeline when user is subscribed to 500 accounts which has X entries per hour.</li></ul><p>Evaluation and experimentation might contain such critical cases until you are confident that a database will be able to serve your performance requirements. A similar thumb of rule is also considering this breakdown when collecting latency metrics and setting SLOs.</p><p>Be careful about high cardinality when collecting metrics per operation. Use logs, even collection or distributed tracing if you need high cardinality debugging data. See <a href="https://medium.com/observability/want-to-debug-latency-7aa48ecbe8f7">Want to Debug Latency?</a> for an overview on latency debugging methodologies.</p><h4>Nested transactions can be harmful.</h4><p>Not every database supports nested transactions but when they do, nested transactions may cause surprising programming errors that are not always easy to identify until it becomes clear that you are seeing anomalies.</p><p>If you’d like to avoid nested transactions, client libraries can do work to detect and avoid nested transactions. If you can’t avoid them, you have to pay attention to avoid ending up surprising situations where committed transactions are accidentally aborted due to a child transaction.</p><p>Encapsulating transactions in different layers can contribute to surprising nested transaction cases and from a readability point-of-view, it might be hard to understand the intend. Take a look at the following program:</p><pre>with newTransaction():<br>    Accounts.create(&quot;609-543-222&quot;)</pre><pre>    with newTransaction():<br>        Accounts.create(&quot;775-988-322&quot;)<br>        throw Rollback();</pre><p>What’s going to be the result of the code above? Is it going to rollback both of the transactions or only the inner one? What happens if we were relying on multiple layers of libraries that were encapsulating the transaction creation from us. Would we be able to identify and improve such cases?</p><p>Imagine a data-layer with several operations (e.g. newAccount) already is implemented in their own transactions. What happens when you run them in higher level business logic that runs in it own transaction? What would be the isolation and consistency characteristics would be?</p><pre>function newAccount(id string) {<br>   with newTransaction():<br>       Accounts.create(id)<br>}</pre><p>Instead of dealing with such open-ended questions, avoid nested transactions. Your data layer can still implement high level operations without creating their own transactions. Then, business logic can start transactions, run the operations on the transaction, commit or abort.</p><pre>function newAccount(id string) {<br>    Accounts.create(id)<br>}</pre><pre>// In main application:</pre><pre>with newTransaction():<br>    // Read some data from database for configuration.<br>    // Generate an ID from the ID service.<br>    Accounts.create(id)</pre><pre>    Uploads.create(id) // create upload queue for the user.</pre><h4>Transactions shouldn’t maintain application state.</h4><p>Application developers might want to use application state in transactions to update certain values or tweaks the query parameters. One critical thing to consider is to having the scope right. Clients often retry the transactions when networking issues happen. If a transaction is relying on state that is mutated elsewhere, it might pick the wrong value depending on the possibility of the data races in the problem. Transactions should be careful about in-application data races.</p><pre>var seq int64</pre><pre>with newTransaction():<br>     newSeq := atomic.Increment(&amp;seq)<br>     Entries.query(newSeq)</pre><pre>     // Other operations...</pre><p>The transaction above will increase the sequence number each time it runs regardless of its end result. If commit fails due to network, on the second retry, it will query with a different sequence number.</p><h4>Query planners can tell about databases.</h4><p>Query planners determine how your query is going to be executed in the database. They also analyze the queries and optimize them before running. Planners can only provide some possible estimations based on signals it has. How to tell how to find the results for the following query:</p><pre>SELECT * FROM articles where author = &quot;rakyll&quot; order by title;</pre><p>There are two ways to retrieve the results:</p><ul><li><strong>Full table scan</strong>: We can go through every entry on the table and return the articles where author name is matching, then order.</li><li><strong>Index scan</strong>: We can use an index to find the matching IDs, retrieve those rows and then order.</li></ul><p>Query planner’s role is to determine which strategy is the best option. Query planners have limited signals about what they can predict and might result in poor decisions. DBAs or developers can use them to diagnose and fine tune poorly performing queries. New releases of databases can tweak query planners and self-diagnosing them can help you when upgrading your database if new version introduces performance problems. Reports such as the slow query logs, latency problems, or stats on execution times could be useful to determine the queries to optimize.</p><p>Some metrics the query planner provides could be noisy, especially when it estimates latency or CPU time. As a supplement to query planners, tracing and execution path tools can be more useful to diagnose these issues even though not every database provides such tools.</p><h4>Online migrations are complex but possible.</h4><p>Online, realtime or live migrations mean migrating from one database to another without downtime and compromising data correctness. Live migrations are easier if you you are migrating to the same database/engine, but can get more complicated when migrating to a new database with different performance characteristics and schema requirements.</p><p>There are different models when it comes to online migrations, here is <a href="http://www.aviransplace.com/2015/12/15/safe-database-migration-pattern-without-downtime/#ixzz3vsEunxmA">one</a>:</p><ul><li>Start doing dual writes to both databases. At this stage, new database won’t have all the data but will start seeing the new ones. Once you are confident about this step, you can move on to the second.</li><li>Start enabling the read path to use both databases.</li><li>Use the new database primarily for reads and writes.</li><li>Stop writing to the old database although keep reading from the old database. At this point, new database still doesn’t have all the new data and you might need to fallback to the old database for old records.</li><li>At this point, old database is read-only. Backfill the new database with the missing data from the old database. Once migration is complete, all the read and write paths can use the new database and the old database can be removed from your system.</li></ul><p>If you need more caste studies, see Stripe‘s comprehensive <a href="https://stripe.com/blog/online-migrations">article</a> on their migration strategy that follows this model.</p><h4>Significant database growth introduces unpredictability.</h4><p>Database growth makes you experience unpredictable scale issues. The more we know about the internals of our databases, the less we might predict how they might scale but there are things we can’t predict.</p><p>With growth, previous assumptions or expectations on data size and network capacity requirements can become obsolete. This is when large scheme rewrites, large-scale operational improvements, capacity issues, deployment reconsiderations or migrating to other databases happen to avoid outage.</p><p>Don’t assume knowing a lot about the internals of your current database is the only thing you need, scale will introduce new unknowns. Unpredictable hotspots, uneven distribution of data, unexpected capacity and hardware problems, ever growing traffic and new network partitions will make you reconsider your database, your data model, your deployment model and the size of your deployment.</p><p>—</p><p>When I mentioned about potentially publishing this article, I already had five more items on my initial draft. Then, I received an overwhelming amount of new <a href="https://twitter.com/rakyll/status/1249771259023392768">ideas</a> on what else to capture. I tried keep the scope limited to the least obvious problems that need the most attention. This doesn’t mean I won’t get to write more on this topic and won’t keep updating this document.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d0178464f78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Persistent Disks and Replication]]></title>
            <link>https://medium.com/google-cloud/persistent-disks-and-replication-9b9412fd9565?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b9412fd9565</guid>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Mon, 21 Oct 2019 02:48:03 GMT</pubDate>
            <atom:updated>2020-05-11T22:55:29.723Z</atom:updated>
            <content:encoded><![CDATA[<p>If you are a cloud user, you probably have seen how unconventional storage options can get. This is even true for disks you access from your virtual machines. There are not many ongoing conversations or references about the underlying details of core infrastructure. One such lacking conversation is how fundamentally <a href="https://cloud.google.com/persistent-disk/">persistent disks</a> and replication works.</p><h4>Disks as services</h4><p>Persistent disks are NOT local disks attached to the physical machines. Persistent disks are networking services and are attached to your VMs as network block devices. When you read or write from a persistent disk, data is transmitted over the network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2IdI1smlXG6YF9PlsT425w.jpeg" /><figcaption>Persistent disks are using Colossus for storage backend.</figcaption></figure><p>Persistent disks heavily rely on Google’s file system called <em>Colossus</em>. Colossus is a distributed block storage system that is serving most of the storage needs at Google. Persistent disk drivers automatically encrypt your data on the VM before it goes out of your VM and transmitted on the network. Then, Colossus persists the data. Upon a read, the driver decrypts the incoming data.</p><p>Having disks as a service is useful in various cases:</p><ul><li>Resizing the disks on-the-fly becomes easier. Without stopping a VM, you can increase the disk size (even abnormally).</li><li>Attaching and detaching disks becomes trivial. Given disks and VMs don’t have to share the same lifecycle or to be co-located, it is possible to stop a VM and boot another one with its disk.</li><li>High availability features like replication becomes easier. The disk driver can hide replication details and provide automatic write-time replication.</li></ul><h4>Disk latency</h4><p>Users often wonder the latency overhead of having disks as a networking service. There are various benchmarking tools available. Below, there are a few reads of 4 KiB blocks from a persistent disk:</p><pre>$ ioping -c 5 /dev/sda1<br>4 KiB &lt;&lt;&lt; /dev/sda1 (block device 10.00 GiB): time=293.7 us (warmup)<br>4 KiB &lt;&lt;&lt; /dev/sda1 (block device 10.00 GiB): time=330.0 us<br>4 KiB &lt;&lt;&lt; /dev/sda1 (block device 10.00 GiB): time=278.1 us<br>4 KiB &lt;&lt;&lt; /dev/sda1 (block device 10.00 GiB): time=307.7 us<br>4 KiB &lt;&lt;&lt; /dev/sda1 (block device 10.00 GiB): time=310.1 us<br>--- /dev/sda1 (block device 10.00 GiB) ioping statistics ---<br>4 requests completed in 1.23 ms, 16 KiB read, 3.26 k iops, 12.7 MiB/s<br>generated 5 requests in 4.00 s, 20 KiB, 1 iops, 5.00 KiB/s<br>min/avg/max/mdev = 278.1 us / 306.5 us / 330.0 us / 18.6 us</pre><p>GCE also allows attaching <a href="https://cloud.google.com/compute/docs/disks/#localssds">local SSDs</a> to virtual machines in cases where you have to avoid a longer roundtrip. If you are running a cache server or running large data processing jobs where there is an intermediate output, local SSDs would be your choice. Unlike persistent disks, data on local SSDs are NOT persistent and will be flushed each time your virtual machine restarts. It is only suitable for the optimization cases. Below, you can see the latency observed from 4 KiB reads from an NVMe SSD:</p><pre>$ ioping -c 5 /dev/nvme0n1<br>4 KiB &lt;&lt;&lt; /dev/nvme0n1 (block device 375 GiB): time=245.3 us(warmup)<br>4 KiB &lt;&lt;&lt; /dev/nvme0n1 (block device 375 GiB): time=252.3 us<br>4 KiB &lt;&lt;&lt; /dev/nvme0n1 (block device 375 GiB): time=244.8 us<br>4 KiB &lt;&lt;&lt; /dev/nvme0n1 (block device 375 GiB): time=289.5 us<br>4 KiB &lt;&lt;&lt; /dev/nvme0n1 (block device 375 GiB): time=219.9 us<br>--- /dev/nvme0n1 (block device 375 GiB) ioping statistics ---<br>4 requests completed in 1.01 ms, 16 KiB read, 3.97 k iops, 15.5 MiB/s<br>generated 5 requests in 4.00 s, 20 KiB, 1 iops, 5.00 KiB/s<br>min/avg/max/mdev = 219.9 us / 251.6 us / 289.5 us / 25.0 us</pre><h4>Replication</h4><p>When creating a new disk, you can optionally choose to replicate the disk within region. Persistent disks are replicated for higher availability. Replication happens within a region between<em> two zones.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*6wLIN8FRBfm7x_G78jf5Zw.png" /><figcaption>Persistent disks can optionally be replicated.</figcaption></figure><p>It is unlikely for a region to fail altogether but zonal failures happen more commonly. Replicating within the region at different zones is a good bet from the availability and disk latency perspective. If both replication zones fail, it is considered a region-wide failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MKgWtAVq-dQsDOXU771v2Q.jpeg" /><figcaption>Disk is replicated in two zones.</figcaption></figure><p>In the replicated scenario, the data is available in the local zone (us-west1-a) which is the zone the virtual machine is running in. Then, the data is replicated to another Colossus instance in another zone (us-west1-b). At least one of the zones should be the same zone the VM is running in.</p><p>Please note that, persistent disk replication is only for the high availability of the <em>disks</em>. Zonal outages may also affect the virtual machines or other components that might cause outage.</p><h4>Read/write sequences</h4><p>The heavy work is done by the disk driver in the user VM. As a user, you don’t have to deal with the replication semantics, and can interact with the file system the usual way. The underlying driver handles a sequence for reading and writing.</p><p>When a replica falls behind and replication is incomplete, read/writes go through the trusted replica until a reconciliation happens. Otherwise, we are in the full replication mode.</p><p>In the full replication mode:</p><ul><li>When writing, a write request tries to write to both replicas and acknowledge when both writes succeed.</li><li>When reading, read request is sent to the local replica at the zone the virtual machine is running in. If the read request times out, another read request is sent to the remote replica.</li></ul><p>—</p><p>It is not always obvious to the users that persistent disks are network storage devices. They enable many use cases and functionality in terms of capacity, flexibility and reliability conventional disks cannot provide.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b9412fd9565" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/persistent-disks-and-replication-9b9412fd9565">Persistent Disks and Replication</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Health, Availability, Debuggability]]></title>
            <link>https://medium.com/observability/health-availability-debuggability-5b0ab300b35c?source=rss-1737b4e67578------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b0ab300b35c</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Thu, 09 May 2019 19:10:25 GMT</pubDate>
            <atom:updated>2019-05-09T19:10:25.432Z</atom:updated>
            <content:encoded><![CDATA[<p>As we collect various observability signals from systems, it fosters a new conversation around the classification of the signals.</p><h3>Jaana B. Dogan on Twitter</h3><p>In observability, we need a new categorization of signals: availability and debuggability signals. Examples of availability signals are metrics, metrics derived from non-downsampled other signals. Debuggability is about logging, tracing, events, etc.</p><p>There is a significant discussion on observability signals and even strong advocacy for one signal over the other. Metrics, events, logs, traces or others? In order to provide some structure to the conversation, it might be productive to provide a high-level breakdown of the signals based on how we utilize them.</p><p>There are three main high-level aspects observability enables:</p><ol><li>Health</li><li>Availability</li><li>Debuggability</li></ol><p>Even though I introduce this classification, it is possible for a single signal to fit all three. It is not the ideal option but imagine how people rely on watching their logs to see a service is up. They are utilizing logs to measure latency for certain operations, and dig into logs to see what’s going on when there is an issue. Even though logs can fit into all three categories, it requires distinct design and planning to make them useful in these categories. And although it is possible to utilize the same signal for all three aspects, we may rely on different signals for each because each signal type has its pros/cons.</p><h4>Health</h4><p>Health checking is fundamentally critical for reliability. We previously wrote on what it means for a system to be <a href="https://medium.com/observability/health-and-availability-b2f32ce28716">healthy</a>. Health is to describe the existence of a system and can be extended to answer its availability to accept work. “Is my server running?”, “Is my worker up?”, “Is my server ready to accept requests?” are the questions you can ask to health signals.</p><p>Health signals are critical for orchestrators such as schedulers and load balancers. Health signals are queried pretty often. They often have to be cheap to produce and easy to query.</p><p>One example of health signals is a health check endpoint served from long-running processes (typically servers):</p><pre>GET /health<br>ok</pre><p>Health signals should be made available quickly at startup. Faster health checking means less overhead for rescheduling, autoscaling, and crash recovery.</p><h4>Availability</h4><p>Availability is the systematic approach to be able to tell if a system is running as users expect. Availability is often quantified and measured by indicators (see <a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/">SLIs</a> as an example formal definition). Then, a target value or range of values (<a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/#objectives-g0s1tdcz">SLOs</a>) are determined for each indicator to represent the acceptable values.</p><p>A system is having an outage when the values are out of the target value/range for the target amount of user requests. And example SLO, “95% of the requests will complete in 100ms.”, will trigger an alert when more than 5% of the requests have a latency longer than 100ms.</p><p>Availability signals are crucial for reliability because they answer the fundamental question of whether a system is working as expected as user wants. This is why availability signals need to report all instances of tasks. The instrumentation should be cheap enough so it is not downsampled. This is why often aggregated metrics are the preferred availability signal. They are compact, and cheap to collect and transfer on wire.</p><h4>Debuggability</h4><p>Debuggability comes into the play mainly for troubleshooting scenarios. Debuggability is also a major aspect in having self-descriptive systems. The more debuggable a system is, the more you can learn about it without having to rely on documentation and reading the codebase.</p><p>Debuggability signals can be considered more optional compared to health and availability signals because they come into play once you figure out the health and availability. Think about signals like logs and distributed traces. They are often more verbose and more expensive to collect. Downsampling the collection is a common strategy to deal with the cost in production. Dynamically enabling more collection and vice versa is also common. You might want to temporarily turn on some collection, investigate an issue and turn it off again.</p><p>—</p><p>Regardless of how you utilize the existing signals available to you from your systems, it is a more healthy approach to think about their role in these high level aspects. Then, you can easily design towards achieving health checking, availability and debuggability based on what signals you can possibly collect. A signal signal might be useful in multiple categories as well as you may need to invest into multiple signals.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b0ab300b35c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/observability/health-availability-debuggability-5b0ab300b35c">Health, Availability, Debuggability</a> was originally published in <a href="https://medium.com/observability">Observability+</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>