DEV Community: Regis Wilson

Development vs Staging vs Production: What's the Difference?

Regis Wilson — Mon, 21 Aug 2023 22:25:52 +0000

The lines between development, staging, and production environments are often blurred. The distinctions may vary depending on many factors, including:

the scale of the organization,
the codebase, or
whether you're viewing the environment from a product, unit testing, or security standpoint.

In this post, I use interviews with fellow developers to understand each environment's purpose and how it's distinct from the others. It's particularly challenging to differentiate between the development and staging environments, and some organizations forgo the staging environment altogether. Let’s find out why, and when it makes sense.

What Is a Development Environment?

Generally, the development environment is the first environment developers use to check if all code changes work well with each other. It's known as a “sandbox” for developers to work in. Examples of commonly used integrated development environments (IDEs) are Visual Studio Code, Eclipse, JetBrains tools, and many others. Note that historically development environments were based on a developer's laptop—a local machine, but with the emergence of cloud, on-demand computing and ephemeral environments, those environments are now being deployed in the cloud.

IDE is where developers’ workflow with code takes place, reloading and debugging aids are enabled here. Also, this environment is where developers can make necessary code changes. In the IDE, approved code can merge with code from other developers working on the same project.

Developers commonly use this space to experiment and receive feedback on what improvements they can make to their work. Consequently, this environment is the most unstable. It's also the most susceptible to bugs and potentially broken code. But, on the upside, in allowing mistakes to happen, this is the most conducive environment to learn collaboratively and create a standardized process.

Besides the most commonly known local machine, there are virtual and cloud-based development environments. Your team might use the virtual and cloud-based environments mainly depending on whether multiple platforms and machines are needed to effectively test and run the code they are writing.

Development environments historically only include a small subset of the entire application and often would lack elements like security, 3rd party APIs and cloud native services. Those would typically be introduced later in the development process and tested in staging. The result, however, turns into frequent rollbacks and bottlenecks in staging. To enable better code quality in development and more frequent release cycles, companies like Release came up with ephemeral environments, a production-like replica that allows developers to properly test their code (i.e shift-left) and isolate bugs to a single branch, while ensuring a smooth merge to staging and production.

What Is a Staging Environment?

Staging environment is the environment where your code is 'staged' prior to being run in front of users so you can ensure it works as designed. The staging environment should mirror production as much as possible. It reflects a production environment not yet exposed to clients, customers, and the general public.

This environment is primarily used for System Integration Testing (SIT) and in-depth manual testing conducted before the client receives code changes. Developers also perform quality assurance (QA), security testing, chaos testing, alpha testing, beta testing, and end-to-end (E2E) testing in this environment.

Additionally, User acceptance testing (UAT) often happens here. In UAT, users can test changes they requested before the new code goes to a production environment.

How you carry out testing in the staging environment can depend on what programming language you're using. For example, Ruby on Rails doesn't have a mode for staging. Rails developers switch modes to a test environment that they use to run testing tools and debug failures. Technically, the Rails Guide delves into how to customize configurations and initialization on applications.

Development vs. Staging Environments

So, now that you know what development and staging environments are, you're probably wondering if you need both. Ultimately, the answer depends on the size of your organization, appetite for risk and speed of change, and your position on making a tradeoff between slowing down the process for quality and testing versus launching new features quickly.

Sometimes smaller companies start out with fewer environments. One developer shared, “You just end up with multiple environments as the organization scales up.”

In some cases organizations with fewer users don't have staging environments. As another developer elaborated:

"Instead, we can deploy in a way that 1% of the traffic will go to each one branch and main branch. Then, we can check the monitoring to see if there are differences between the two. When we are certain that at the most we will affect 1% of traffic and everything is fine, we will then proceed with merging the two branches. I think it would be ideal if the continuous integration (CI) and continuous deployment (CD) process were to set up that 1%, then we could verify the results. This is the same as I have seen for verifying front-end changes in continuous integration."

Is a Staging Environment Necessary?

Deploying to staging is safe, because it will not affect users, but is not necessarily effective because you might not test all the features or combinations that end users will be using. The general solution to this problem is to deploy to production as quickly as you can but only enable or test subsets of new features with flags or canary testing. This way you are only risking challenges for a small subset of users, and are able to see the application perform with live traffic in the production environment.

Developers say they like to see how real traffic works through the codebase and compare this technique to feature flagging. This may eliminate the need for a beta environment. This results in the concept of "staging" not being a distinct environment.

However, developers agree that it's useful to have a separate beta domain to make significant changes. According to Atlassian CI/CD, feature flagging allows developers to turn functionality on and off during runtime. That way, you don't need to deploy code at every update.

What Is a Production Environment?

The production environment is the live site complete with performance optimizations. The codebase must be secure, performant, stable, and able to sustain heavy traffic as the client, customers, and public use it.

There is a common misconception that production is more important than development or staging. Actually, the reverse could be true: development environments could be so critical to the business that they cannot tolerate any downtime at all but production can tolerate some downtime.

As an example at Truecar and in most other companies I worked at, the website could be broken for some amount of time as long as it came back up relatively quickly. However, if development was down for more than an hour, you could be looking at losing an entire day of developer features for the whole company!

Regardless of your setup, you should treat production with care, and restrict who updates the production code. Ideally, you won't be building new versions of the codebase for the production environment; it's better to deploy the same builds to the staging environment.

At this point of the software development lifecycle, the code shouldn't have any bugs or require fixes. To avoid a poor user experience, you should consider it the final product.

However, you can make urgent fixes in the production environment if needed. In doing so, you can consistently improve upon quality control for product releases, making it easier to keep tabs on new product updates.

Conclusion

Although the development, staging, and production environments converge, they have their own significance in the larger software development lifecycle. The significance of each environment depends on the organization running the system.

The way a company treats and leverages these environments today differs wildly depending on the organization and its DevOps practices and policies. Sometimes teams within the same organization use these environments in different ways and have different philosophies of what they mean, and how critical they are to the company’s mission.

From my conversations with individuals who play different roles in the tech industry, I can say the overall development culture is shifting progressively toward promoting new code to all these environments as soon as possible. One developer expressed,

"The idea is that even the smallest code change gets released to production in a matter of minutes, not months."

With that in mind, the common goal is that the folks responsible for the software development life cycle want more efficient environments for producing the highest quality codebases. These people continuously strive to find new methods to make that process easier.

For a better understanding of what environments are and to be inspired about optimizing them, read more about staging environments, ephemeral environments, and UAT with Release ephemeral environments.

Using Docker Build and BuildX to Host Production Websites in Kubernetes

Regis Wilson — Fri, 04 Aug 2023 22:16:26 +0000

Introdction

Docker is all the rage these days and hosting containers in production is becoming the norm. Complex application stacks have evolved beyond the t2.small instances running a LAMP stack in AWS. But what are you going to do if your successful business is built on a wonderful PHP shopping site and you are expected to learn all the ins and outs of deploying and orchestrating images on today's advanced platforms? As each day passes, more and more of our t2 instances running RHEL5 and RHEL6 are being torn down by AWS and not able to be replaced by any suitable alternatives.

This guide will help you avoid orchestration and deployment woes by hosting your production PHP application using docker build and a new easy-mode entry into Kubernetes with docker bake. The examples presented will show a modern PHP7 stack running on RHEL8 using the latest innovations available to production workloads today. You do not need to learn complex ideas about docker push, docker run, kubectl, and more. With this example you can be up and running in the modern era in as little as 30 minutes.

Starting Simple with Docker Build

docker build is a beginners' command that allows you to execute complex workflows and shell scripts inside a container without complex infrastructure or expertise. If you are new to docker build, you can find some good resources online. Basically, you would create a Dockerfile and then provide instructions on which OS to choose, which yum packages to install, copy your code into the directories, and then run your website commands from there.

This example is based on the excellent blog post by RedHat.

Save the following example in your CVS folder where your PHP code lives and name it Dockerfile.rhel8.

FROM registry.access.redhat.com/ubi8/ubi:8.1 AS runtime

ARG NGROK_TOKEN

#RUN yum --disableplugin=subscription-manager -y module enable php:7.3 \
#  && yum --disableplugin=subscription-manager -y install httpd php wget \
#  && yum --disableplugin=subscription-manager clean all
RUN yum --disableplugin=subscription-manager -y install httpd php wget \
  && yum --disableplugin=subscription-manager clean all

RUN wget --no-verbose 'https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz' \
  && tar xvzf ngrok-v3-stable-linux-amd64.tgz -C /usr/local/bin \
  && rm -f ngrok-v3-stable-linux-amd64.tgz

COPY *.php /var/www/html

RUN sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf \
  && sed -i 's/listen.acl_users = apache,nginx/listen.acl_users =/' /etc/php-fpm.d/www.conf \
  && mkdir /run/php-fpm \
  && chgrp -R 0 /var/log/httpd /var/run/httpd /run/php-fpm \
  && chmod -R g=u /var/log/httpd /var/run/httpd /run/php-fpm

FROM runtime

COPY *.php /var/www/html

RUN php-fpm & httpd -D FOREGROUND & NGROK_TOKEN=$NGROK_TOKEN ngrok http 8080 --log=stdout

This file will create a docker container and run all of the setup commands you need to install the full LAMP stack you are used to without any infrastructure or devops help. In each step, we execute setup commands and download yum packages to add software to your running container.

You will notice that although docker build is very powerful and can run any command you tell it to, it will not allow you to connect to any network resources for safety and security. That is, normally, you could just run nginx and then visit http://localhost:8080 to visit your website. The nginx command is running on localhost but you will not be able to access it. It is a different localhost inside the container. You could call it remotehost in this scenario.

All of these networking details are securely hidden by the safety precautions of docker build. You will use these safety features when you get to deploy to production because the networking features are not available with the docker build commands. So we use ngrok to go around these details.

The ngrok tunnel command can be customised to use a custom domain on the endpoint you select and where you issue the correct ngrok token. To run your website locally, simply type this on your laptop (this is not production yet, so you can use an auto-generated hostname at this point):

read -p "Input token? " NGROK_TOKEN
export NGROK_TOKEN
docker build -f Dockerfile.rhel8 -t mywebsiteapp:1 --build-arg NGROK_TOKEN=$NGROK_TOKEN .

This will ask for your production ngrok token for hosting and then your website will be available:

Input token? *********
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile.rhel8
#2 transferring dockerfile: 940B done
#2 DONE 0.0s

#3 [internal] load metadata for registry.access.redhat.com/ubi8/ubi:8.1
#3 DONE 0.5s

#4 [1/6] FROM registry.access.redhat.com/ubi8/ubi:8.1@sha256:1f0e6e1f451ff020b3b44c1c4c34d85db5ffa0fc1bb0490d6a32957a7a06b67f
#4 DONE 0.0s

#5 [2/6] RUN yum --disableplugin=subscription-manager -y module enable php:7.3   && yum --disableplugin=subscription-manager -y install httpd php wget   && yum --disableplugin=subscription-manager clean all
#5 CACHED

#6 [internal] load build context
#6 transferring context: 30B done
#6 DONE 0.0s

#7 [3/6] RUN wget --no-verbose 'https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz'   && tar xvzf ngrok-v3-stable-linux-amd64.tgz -C /usr/local/bin && rm -f ngrok-v3-stable-linux-amd64.tgz
#7 3.511 2023-08-04 21:41:43 URL:https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz [8826207/8826207] -> "ngrok-v3-stable-linux-amd64.tgz" [1]
#7 3.519 ngrok
#7 DONE 3.8s

#8 [4/6] ADD index.php /var/www/html
#8 DONE 0.1s

#9 [5/6] RUN sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf   && sed -i 's/listen.acl_users = apache,nginx/listen.acl_users =/' /etc/php-fpm.d/www.conf   && mkdir /run/php-fpm   && chgrp -R 0 /var/log/httpd /var/run/httpd /run/php-fpm   && chmod -R g=u /var/log/httpd /var/run/httpd /run/php-fpm
#9 DONE 0.5s

#10 [6/6] RUN php-fpm & httpd -D FOREGROUND & NGROK_TOKEN=xyzzy ngrok http 8080 --log=stdout
#10 0.586 t=2023-08-04T21:41:44+0000 lvl=info msg="no configuration paths supplied"
#10 0.586 t=2023-08-04T21:41:44+0000 lvl=info msg="ignoring default config path, could not stat it" path=/root/.config/ngrok/ngrok.yml
#10 0.587 t=2023-08-04T21:41:44+0000 lvl=info msg="starting web service" obj=web addr=127.0.0.1:4040 allow_hosts=[]
#10 0.601 AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
#10 0.941 t=2023-08-04T21:41:44+0000 lvl=info msg="client session established" obj=tunnels.session obj=csess id=c8fe54e8acb1
#10 0.941 t=2023-08-04T21:41:44+0000 lvl=info msg="tunnel session started" obj=tunnels.session
#10 1.050 t=2023-08-04T21:41:44+0000 lvl=info msg="started tunnel" obj=tunnels name=command_line addr=http://localhost:8080 url=https://999-99-999-9-99.ngrok.io

At this point your website is running and you will be able to get your browser to load your new website in your browser:

Notice that you do not need to use any complicated push, pull, or run commands. The container simply runs and waits for you to test your code.

I Feel the Ache, the Cache for Speed

You may notice that Docker will not cache the final results for your build because it does not complete a full layer. This is by design so that your code is secured and not stored anywhere that is unsafe. However, by utilising the multi-stage dockerfile available to the most elite Docker users, you can see that subsequent builds will be very fast, slowing down only to copy your code base after it is updated or when you restart the development environment.

$ docker build -f Dockerfile.rhel -t rhel8:4 . --build-arg NGROK_TOKEN=$NGROK_TOKEN --progress plain
#1 [internal] load build definition from Dockerfile.rhel
#1 transferring dockerfile: 966B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for registry.access.redhat.com/ubi8/ubi:8.1
#3 DONE 0.3s

#4 [runtime 1/4] FROM registry.access.redhat.com/ubi8/ubi:8.1@sha256:1f0e6e1f451ff020b3b44c1c4c34d85db5ffa0fc1bb0490d6a32957a7a06b67f
#4 DONE 0.0s

#5 [runtime 2/4] RUN yum --disableplugin=subscription-manager -y module enable php:7.3   && yum --disableplugin=subscription-manager -y install httpd php wget   && yum --disableplugin=subscription-manager clean all
#5 CACHED

#6 [runtime 3/4] RUN wget --no-verbose 'https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz'   && tar xvzf ngrok-v3-stable-linux-amd64.tgz -C /usr/local/bin && rm -f ngrok-v3-stable-linux-amd64.tgz
#6 CACHED

#7 [internal] load build context
#7 transferring context: 30B done
#7 DONE 0.0s

#8 [runtime 4/4] RUN sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf   && sed -i 's/listen.acl_users = apache,nginx/listen.acl_users =/' /etc/php-fpm.d/www.conf   && mkdir /run/php-fpm   && chgrp -R 0 /var/log/httpd /var/run/httpd /run/php-fpm   && chmod -R g=u /var/log/httpd /var/run/httpd /run/php-fpm
#8 DONE 0.4s

#9 [stage-1 1/2] COPY index.php /var/www/html
#9 DONE 0.1s

#10 [stage-1 2/2] RUN php-fpm & httpd -D FOREGROUND & NGROK_TOKEN= ngrok http 8080 --log=stdout
#10 0.477 t=2023-08-05T22:33:54+0000 lvl=info msg="no configuration paths supplied"
#10 0.477 t=2023-08-05T22:33:54+0000 lvl=info msg="ignoring default config path, could not stat it" path=/root/.config/ngrok/ngrok.yml
#10 0.478 t=2023-08-05T22:33:54+0000 lvl=info msg="starting web service" obj=web addr=127.0.0.1:4040 allow_hosts=[]
#10 0.487 AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
#10 0.816 t=2023-08-05T22:33:54+0000 lvl=info msg="client session established" obj=tunnels.session obj=csess id=88117dc8c891
#10 0.816 t=2023-08-05T22:33:54+0000 lvl=info msg="tunnel session started" obj=tunnels.session
#10 0.906 t=2023-08-05T22:33:54+0000 lvl=info msg="started tunnel" obj=tunnels name=command_line addr=http://localhost:8080 url=https://abcde-99-999-9-99.ngrok.io

You can see how blazingly fast the steps are now that the previous layers have been cached and your code is safe and secure in memory. The previous build took over 6 seconds to complete, while the cached version took less than 0.7 seconds, a savings of just under 90%!!

Updating your code locally before running in production

Obviously you will want to make some code changes and adjust and test your site before you are ready to publish to production. No problem, use the following example on in another terminal in your laptop:

cd production-web-site
cvs update
pico sales.php
<make your changes and save>
cvs add sales.php
cvs commit -m "Add thinner border and show postal code" .

Switch back to your original docker build command terminal and hit APPLE-CMD-C to terminate the original site. Then run version two of your site again:

read -p "Input token? " NGROK_TOKEN
export NGROK_TOKEN
docker build -f Dockerfile.rhel8 -t mywebsiteapp:2 .

Running in Kubernetes

Everyone wants to deploy to Kubernetes, but it is difficult, if not impossible. However, it is relatively simple to add a buildx builder to your cluster to deploy production-level quality for your containers. Again, no push and pull and run complexities are required.

This new feature of docker build that we rely on is the ability to deploy running containers to Kubernetes, using the docker build bake commands. The bake commands are an extension of build, much like you would build an oven and then bake a cake. It is the same with the code.

To be able to run your production website on Kubernetes with bake you will first create the buildx instance pod that will run your production website:

$ docker buildx create mywebsite --driver kubernetes --driver-opt nodeselector=kubernetes.io/arch=amd64
watoosi-canal

You can list your builders with the following command

docker buildx ls
NAME/NODE       DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
watoosie-canal* kubernetes
  watoosie-canal0 ......... running v0.19.4  linux/amd64

Next, you will need to create a bake file which is very similar to a recipe for your code to go into the Kubernetes oven.

A quick note on the ngrok configuration: you can supply your credentials to the ngrok.yaml configuration file with hostnames and so forth. This is the recommended way to set the custom domain name, web and security settings, and so forth. You can visit the ngrok documentation for more details.

Simply create a file like this called docker-bake.prod.hcl with the following contents:

target "webapp-release" {
  dockerfile = "Dockerfile.rhel8"
  tags = ["mywebsiteapp"]
  platforms = ["linux/amd64"]
  args = {
    NGROK_TUNNEL = "<insert credentials here>"
  }
}

Then run the following command to connect to Kubernetes, deploy the build container and run your website!

docker buildx bake -f docker-bake.prod.hcl webapp-release

(if your website crashes or you want to restart it, you might need to find the build instance and use it as follows:)

$ docker buildx ls
....
$ docker buildx use <name of instance>

Visit your production website in the browser and you will see something like this:

Conclusion

If you use docker build in production, I'd like to hear about it! We can share the best practices and write more blog posts to share knowledge.

Photo by Josue Isai Ramos Figueroa on Unsplash

Syncing Databases: How to Do It and Best Practices

Regis Wilson — Fri, 16 Jun 2023 00:17:16 +0000

As an engineering leader or developer, you may have encountered the need to sync multiple copies of a database to ensure data consistency across different systems or locations. Whether you're working on a distributed system, a mobile application, or a cloud-based platform, syncing the databases is a crucial task that requires careful planning and execution.

In this article, we'll look at database synchronization and what to use it for. Next, we'll explore different types of synchronization processes and how to sync databases step by step. We'll also cover some helpful tooling that makes syncing your databases easier and faster. Let's get started!

What Is Database Synchronization?

Database synchronization is the process of keeping multiple copies of a database in sync with one another.

There are different ways of synchronizing databases, depending on the type of database, the network infrastructure, and the application's requirements. Some of the standard methods of database synchronization include the following:

Two-way synchronization

In two-way synchronization, both databases can make changes and synchronize with each other.

One-way synchronization

In one-way synchronization, one database acts as the source, and the other databases are updated to match it.

Incremental synchronization

With incremental synchronization, changes are only made since the last synchronization.

File-based synchronization

With file-based synchronization, data is exported to a file and imported into the other databases.

Use Cases for Synchronization

Database synchronization is used in various situations where multiple copies of a database are in use and when it's necessary to ensure that the data in each copy is consistent. Some common use cases include the following:

Backup and recovery

You can use database synchronization to keep a secondary copy of a database in sync with the primary copy, providing a way to recover from data loss or corruption.

Mobile and offline applications

Applications that work offline or on mobile devices may need to synchronize data with a central database when a connection becomes available. Database synchronization ensures that the data on the mobile device or offline application is consistent with the data on the central server.

Collaborative platforms

Multiple users may work on the same data in a collaborative platform. Here, database synchronization ensures that changes made by one user propagate to all other users, maintaining data consistency.

Distributed Systems

In a distributed system, multiple copies of a database may run on different servers or in different locations. Database synchronization ensures that changes made to one copy of the database propagate to all other copies, maintaining data consistency across the system.

Cloud-based systems

Cloud-based systems often have multiple copies of a database running in different regions to provide high availability and reduce latency. Database synchronization ensures that data is consistent across all copies of the database.

Types of Database Synchronization

There are several different types of database synchronization, each with its own advantages and disadvantages. Some common types include the following:

Source/Replica replication

In this type of replication, one database acts as the source, and the other databases are updated to match it. The source database receives all updates and changes, which then propagate to the replica databases. It's commonly used for read-heavy workloads.

Multi-master replication

In this type of replication, all databases can act as both sources and replicas. Changes made to one database are propagated to all other databases, ensuring that all copies of the data are consistent. This type of replication is helpful for write-heavy workloads.

File-based synchronization

In this type of synchronization, data is exported to a file and then imported into the other databases. It's a simple method that's easy to implement, but it can be slow and may not be suitable for large amounts of data.

Log-based replication

In this type of replication, changes made to a database are recorded in a log and then propagated to the other databases. This allows for fast and efficient replication but can be more complex to set up and maintain.

Trigger-based replication

In this type of replication, triggers are set up on the source database to capture changes, which can then be propagated to the target database. It allows for fine-grained control over which changes are propagated, but it can be resource-intensive and may be unsuitable for high-traffic systems.

Cloud-based database synchronization

We can also use cloud services like AWS Database Migration Service (DMS) and Azure Database Migration Service to sync databases. This is a good option if you have a cloud-based infrastructure and want to leverage the scalability and reliability offered by these services.

The best approach for your use case will depend on the type of data, the number of databases, the network infrastructure, and the requirements of the application.

‍How to Sync Databases

The process of syncing databases can vary, depending on the type of databases, the method of synchronization, and the specific requirements of the application. Below is the step-by-step guide on how to sync databases:

Step 1: Understand your use case

Understand the specific requirements of your use case and choose the synchronization method that best fits those needs.

Step 2: Identify the databases to be synced

Determine which databases need to be synced and the type of data they contain.

Step 3: Choose a synchronization method

Decide on the method of synchronization that's most appropriate for your use case. This may be replication, a data syncing tool, a custom script, or a cloud-based service.

Step 4: Configure the databases

Set up the databases for synchronization. This may include configuring replication settings, installing data syncing tools, or writing custom scripts.

Step 5: Test the synchronization

Test the synchronization by making changes to one database and verifying that the changes are propagated to the other databases. This will help you identify any issues or bugs before deploying it in production.

Step 6: Schedule synchronization

Set a schedule for the synchronization to occur regularly. You can do this using a built-in scheduling feature or by writing a custom script.

Step 7: Monitor and troubleshoot

Monitor the synchronization process and troubleshoot any issues that arise. This may include monitoring replication lag, checking for errors, and addressing any conflicts that occur. It may also include monitoring replication lag, checking for errors, and addressing any conflicts that occur.

Step 8: Maintain and update

Regularly maintain and update the synchronization process to ensure that it continues to function correctly.

Note: Some steps may vary, depending on the type of database and the method of synchronization. For example, for a cloud-based service like AWS DMS, the process can be simpler. You have to create the replication task and configure the source and target databases.

Tooling for Database Synchronization

There are various tools available for database synchronization, depending on the type of database and the method of synchronization you're using. You can achieve database synchronization through various methods, such as the following:

Custom scripts

You can write custom scripts using programming languages such as Python or Java to sync databases. This involves writing code to compare data in two databases and making changes as needed.

Replication

This involves copying data from one database to another so that changes made to one are reflected in the other.

MySQL provides built-in replication capabilities, allowing users to replicate data between two or more MySQL servers.
Microsoft SQL Server also provides built-in replication capabilities, including transactional replication and merge replication.
PostgreSQL offers several replication solutions, including streaming replication and logical replication.
MongoDB provides built-in replication features, including replica sets and sharded clusters for horizontal scalability.

Cloud-based services

Below are some of the common cloud-based services that you can use to sync databases.

AWS Database Migration Service (DMS) can migrate, replicate, and sync databases between different platforms and environments, including on-premises environments and cloud-based environments like Amazon Web Services (AWS).
Azure Database Migration Service is a fully managed service designed to enable seamless migrations from multiple database sources to Azure.

Data syncing tools

Various data syncing tools can automate the process of keeping databases in sync.

SymmetricDS is open-source data synchronization software that supports multiple relational databases, including MySQL, PostgreSQL, Oracle, and more.
Talend, Informatica, and Boomi can automate the process of keeping databases in sync.
Oracle GoldenGate is a real-time data integration and replication software for heterogeneous environments, including Oracle, SQL Server, DB2, and more.

These are some of the popular tooling used for database synchronization. However, the best tool depends on the type of database, the method of synchronization, and the specific requirements of the application.

Best Practices

When working with database synchronization, there are several best practices that can help you ensure that your synchronization process is efficient and reliable.

Keep your databases in sync

Regularly check and compare data between the different copies of the database and make updates as needed.

Use a replication tool

Use a replication tool that best fits your use case and the type of data you're working with.

Backups

Regularly back up your databases to ensure that you can recover from data loss or corruption.

Use cloud-based services

You can also use Cloud-based services like AWS DMS and Azure Database Migration Service to sync databases. It's a good option if you have a cloud-based infrastructure and want to leverage the scalability and reliability offered by these services.
‍

Security

Ensure that your synchronization process is secure by encrypting data in transit and at rest and by implementing access controls.

By following these best practices, you can ensure that your databases are kept in sync and that your synchronization process is efficient and reliable. Additionally, it's always important to be aware of the particularities of the database you're working with, the replication tool you're using, and the type of data you're syncing in order to apply the best practices in a way that fits your specific needs.

Conclusion

Syncing databases is a crucial task that requires careful planning and execution. By understanding your use case, choosing the right synchronization method, and following best practices such as testing, monitoring, and maintaining, you can ensure that your databases sync effectively and efficiently. Additionally, by using cloud-based services, you can reduce the complexity of the process and ensure that your databases are always in sync.

Full Fidelity Data for Ephemeral Environments

Regis Wilson — Thu, 09 Mar 2023 17:57:15 +0000

This blog post is a follow-up to the wildly successful webinar on the same topic. Refer to our webinar “Full Fidelity Data for Ephemeral Environments” for more information. In the webinar and this post, we will explore why you should use production-like data in ephemeral environments, how you can (and should) do so, and how to reduce the cost, complexity, and avoid difficulties associated with using production-like data at scale.

Overview

Ephemeral environments are an indispensable tool in almost any software development workflow. By creating environments for testing and pre-production phases of the workflow, advanced software development teams can shift testing and verification “left” (that is, earlier in the life cycle rather than later when features or bugs reach production).

The reason that environments should be ephemeral is that they can be set up on-demand with specific features and/or branches deployed in them and then torn down when the task is completed. Each feature or branch of the development code base can be tested in an isolated environment as opposed to the legacy shared and fixed stages of QA, Staging, User Acceptance Testing (UAT), and so forth.

Ephemeral environments are always used for a specific purpose so it is clear which feature and code base is deployed. The data is ideally based on a very close approximation to production so that the feature under development or test can be deployed into production with as much confidence as possible.

Why Use Production-Like Data in Ephemeral Environments?

The best way to test new features or fix bugs that exist in production is to use the same code and data from the production environment. Version Control Systems (VCS) like git solved this problem decades ago, at least for code. But the data portion has long lagged behind due to complexity in access and cost, which we will address toward the end of this post.

Testing code against stale or seed data is fine for automated testing, but when developing new features or diagnosing problems in production, the data should mimic production as closely as possible. Otherwise, you are risking chasing the same fugs over and over again, or launching a faulty feature.

It is rare that data in production is static and unchanging; if it were, you could make the database read-only! While some amount of read-only or write-rarely data exists, it almost always needs to be updated and changed at some point, the only difference is a matter of frequency of updates.

Because the data is a living and breathing thing that changes and evolves constantly and possibly is updated based on customer live inputs or actions, the legacy strategy of fixed QA, staging, and test databases get out of date extremely quickly. In my experience, a fixed environment will stray from production data in as short a time as a few days or weeks. Many times the QA or staging databases are several years out of date from production, unless you specifically “backport” data from production.

Lastly, production databases and datasets are often quite large (and grow larger every day) compared to fixed QA and staging databases. Thus, testing on limited data or fake seed data when developing new features or changing the code base can introduce unexpected regressions in performance or bugs when large results are pulled out of a table.

Should I Really Use Production-like Data in Ephemeral Environments?

Short answer, yes. However, you need to be mindful of the possible moral, ethical, and legal implications in using actual production data subject to HIPPA or other regulatory controls. This is why it is crucial to generate a so-called Golden Copy of the data that is scrubbed of any private or confidential data, while still maintaining the same qualities as production in terms of size, statistical parameters, and so forth. This Golden Copy is the source of truth that is updated frequently from your actual production data. We recommend daily updates, but depending on your particular use case it can be more or less often.

With the Golden Copy that is sufficiently production-like (each case varies in how much fidelity to production is required), it is possible to accurately portray behavior features as they would occur in production. Transactions, events, customer (faked) accounts, and so forth are all available and up-to-date with real (-ish) data. For example, if there is a bug that manifests in production, an engineer could easily reproduce the bug with a high degree of confidence, and either validate or develop fixes for the problem.

Testing features with fake or fixed data is suitable for automated testing but for many use cases, especially when testing with non-technical users, real production-like data is valuable to ensure the feature not only works properly but also looks correct when it reaches production.

Isn’t that Terribly Difficult, Expensive, and Prohibitive? Or, “My Database is too Unique.”

The most common objections to using a production-like dataset are around the difficulties in creating, managing and accessing the dataset, and around the overall cost. I can try to address these in two points: the first one (creation/maintenance/access) can be pretty difficult depending on your use case, but it can be solved; the second one (cost) can readily be handled in this modern era of cloud computing and on-demand managed services offered by cloud providers.

The first problem is that access to production data is usually kept in the hands of a small group of people, and those people are usually not the software engineers who are developing code and testing new features. Separation of concerns and responsibilities is vital for keeping production data clean and secure, but this causes problems when engineers need to test new features, verify bugs in production, or experiment with changes that need to be made.

The best practice is to generate a Golden Copy of the data that you need as mentioned above. The Golden Copy should be a cleaned, stable, up-to-date version of the data that exist in production, but without any data that could compromise confidentiality or proprietary information if it were to accidentally be exposed, (or even internally exposed).

What I tell most people who are involved with production datasets is to create a culture of welcoming access to the Golden Copy and distributing the data internally on a self-service model so that anyone who would like the latest snapshot can access it relatively easily and without a lot of hoops to jump through. Making the data available will ensure that your cleaning and scrubbing process is actually working properly and that your test data is going to have a high degree of similarity to the data in production. This will only make your production data and operations cleaner and more efficient, I promise you.

The second problem is that allowing software engineers, QA people, testers, and even product people access to these datasets comes at a cost. Every database will typically cost a certain amount of money to run and store the data, but there are definitely some optimisations you can implement to keep costs down while still enjoying access to high quality datasets.

The best way to keep costs down is to make sure that the requested access to the production-like dataset is limited in time. For example, a software engineer might only need to run the tests for a day or a week. At the end of that time, the data should automatically be expired and the instance removed because it is no longer needed. If the data is still needed after the initial period of time has expired, you can implement a way to extend the deadline as appropriate.

Another way to reduce costs is to use Copy on Write (COW) technologies if they are available for your database engine and cloud provider. The way this works is that the Golden Copy holds most of the data in storage for use, while the clones that are handed out to engineers are sharing most of the data with the original. It is only when a change or update is made to a table or row that the data is “copied” over for the clone to use. This is what a Copy on Write will do: it means that the only additional costs for storage on the clone are the incremental changes or writes that are executed during testing.

Another good way to reduce costs is to pause or stop a database when it is not in use. Depending on your cloud provider, you may be able to execute a pause or stop on the database instance so that you can save money during the evenings or weekends when the database is not likely to be in use. This can save 30-60% off your costs versus running the database 24/7.

The good news is that Release offers all of the features, including cloning snapshots from the Golden Copy, pausing and restarting databases on a schedule and expiring them as well, and using COW to save time, money, and storage. We support all of the above (using AWS, GCP and soon Azure) and we can easily build a dataset pipeline that is checked out to each ephemeral environment where your code is deployed.

You can refresh the dataset to get new changes from the Golden Copy (which is a cleaned version of production, or directly from production as you wish), and you can also update the data in your ephemeral environment to start over from scratch with a new database. You can also set an expiration for each ephemeral environment that will last as long as the branch or feature pull request is open, and engineers can extend the environment duration as needed. Lastly, you can set a schedule for pausing the databases in the dataset so that you save additional costs when the environments are unlikely to be used.

Ready to learn more? Watch the on-demand webinar “Full Fidelity Data for Ephemeral Environments.”

About Release
Release is the simplest way to spin up even the most complicated environments. We specialize in taking your complicated application and data and making reproducible environments on-demand.

How to Self-Host PostHog Using Release Part One

Regis Wilson — Wed, 15 Feb 2023 19:58:38 +0000

This first part of a series explains how to run a hobby version of PostHog on your own cloud infrastructure using Release. Check back later to read how to perform the self-hosted enterprise version. You can read the PostHog FAQ for more details on the software and self-hosting options for hobby and enterprise options.

Introduction

PostHog is an open source tool for collecting and analyzing behavior metrics and events from your application without sending data to a third party hosting provider. As the GitHub repository says, “…third-party analytics tools do not work in a world of cookie deprecation, GDPR, HIPAA, CCPA, and many other four-letter acronyms. PostHog is the alternative to sending all of your customers' personal information and usage data to third-parties.”

This goes together with the Release philosophy of deploying high-quality, full-stack environments in your own cloud accounts. Deploying PostHog next to your development environments for complete testing and development, or as a shared staging location for integration testing is a valuable component for developing and testing user interactions and metrics collection that would be much harder to support without Release.

This article will walk you through the simple steps for configuring a PostHog application template that you can deploy to your own cloud infrastructure using Release in about 30 minutes or so. The hobby version is a full functionality installation of PostHog but will not be configured with redundant services and long-term storage and backup solutions like the enterprise version would. The most common use-case for wanting to install the hobby version is to support developer environments for testing in isolation or for QA environments for complete end-to-end testing of product analytics.

We will cover the enterprise version for permanent installations (like staging or production) in a followup post. The most common use-case to install the enterprise version is to self-host and scale your own analytics engine and data from your product customers without sending the data outside your organization or to a third-party Software-as-a-Service (SAAS). Release allows you to self-host applications in your own cloud environments keeping your and your customers’ data safe and secure.

Prerequisites

To get started, open a Release account which allows you to have unlimited applications and pay only for the environments you are running. You can test out Release by creating a trial account in a shared environment; or if you already have a paid plan, you can use your Release account to host these environments in your own cloud infrastructure next to your existing applications. See our quickstart documentation for more details.

Next, fork the PostHog GitHub repo using either our fork with the configuration options already built in, or the original upstream repository. You can also use one of our integrations with GitLab or BitBucket but you will need to clone and push the repository separately before you get started.

Configure the PostHog Application

The first step is to import the application into Release by analyzing the repository and reading in the docker-compose and package.json files to get a running version of the repository in your account. If you are using the fork from our repository, then you will have a head start by using our configuration YAML already integrated into the repository. These instructions will work for the plain upstream version and they show how the process works.

First, click the “Create New App” button and select your forked repository (note: if the repository does not show up, go to your profile page and make sure you have configured the correct permissions and scope for the version control system integration you are using). Give the application a name and go to the next step.

Next, analyze the repository using a branch and the common hint files we search for to configure this application. Select at least one docker-compose file and then select the services to your preference as shown in the following image. The hobby version will not include any helm charts or cloud native service integrations, but this is a good example and low-cost way to test out the functionality of the application.

When you’ve selected the services to import and analyze, you will fine tune the application template so that it has the specific customizations and workflows you would like to use for testing purposes. You can start by editing the hostnames for each service (you will only need a service hostname for web, maildev, and clickhouse for example). You can also customize the domain you will deploy the test application URLs to.

You should pay close attention to the workflow steps as the order of the services is important. Reading up on the architecture diagram and understanding the depends_on fields in the docker compose will help you understand the correct workflow as well. This is all organized for you in the forked version we host and maintain.

Once you are happy with the application template and service definitions, you can proceed with fine-tuning the environment variables for the application. Here, the PostHog documentation is excellent in helping you compare the defaults and necessary customization you will need side-by-side.

You can see that I’ve set some environment variable mappings which allow interpolation of Release-injected variables so that the application URLs and internal configuration items work correctly for each environment that will be created. Since we are using a hobby (or ephemeral) version of this application, we are not going to hardcode values or secrets as we might do in a permanent or production environment.

The next steps involve creating build arguments. We always recommend using the production value for NODE_ENV since Release ephemeral environments run in production mode, rather than development mode (except when using our Remote Development feature).

When those steps are done, you can go back to review all your work and confirm it looks good or you can immediately deploy your environment and see the results of your hard work! The application template and environment variables will be deployed into your cloud environment in a completely isolated environment that you can test and play with. When you are done with testing the environment, it can be expired, removed, or you can redeploy it with any changes and tweaks you need to make until you are satisfied with the results.

Deploy the Application as an Environment

You should see your deploy starting up and an ephemeral environment is immediately spun up to begin playing with!

‍
‍
Once complete, you can visit the environment status page which shows you a list of clickable URL links for your application (we recommend editing those down to services that you actually need, for example, redis and postgres are not going to be reachable on the internet and you should remove those from your configuration).

Debug and Test the Environment

You can inspect the services and statuses of each underlying instance that is running in your kubernetes namespace. This gives you the up-to-date picture of what is started and what is running properly, what needs to be investigated for errors, logs, and so-forth. Alternatively, you can jump into a terminal session on a service for immediate debugging and testing.

Clicking on each service link, you can see Clickhouse works

And the same with maildev:

Login and Configure the Application

But the most important thing to look at is the PostHog application itself. We will click on “Just experimenting” for now.

We can verify that all the services and prerequisites are running:

Then get started by creating an account:

And finally you can start configuring your application and settings to use in your new isolated hobby experience:

Conclusion

Using Release to deploy a hobby version of PostHog allows you to run an isolated version of analytics and metrics collection during the development and testing of your own application code. PostHog can be deployed alongside every application deployment you use for each developer or for each branch/feature/pull request that needs to integrate with PostHog, depending on your workflow and requirements.

With Release, it is easy to spin up environments for almost any conceivable use case and this empowers developer teams to deploy applications without relying on a devops or an infrastructure team and minimizing costs associated with provisioning and maintaining the application. You can deploy dependencies or third-party tools that need to be tested or verified when performing application changes or when rolling out new features. Verifying this functionality with third party tools and services is a valuable way to ensure that features reaching staging or production will work even during the earliest phases of development.

Ready to get started? Open a Release account and start your free trial today or check out our quickstart documentation for additional information.

Kubernetes Pod: A Beginner's Guide to an Essential Resource

Regis Wilson — Tue, 13 Sep 2022 20:11:27 +0000

Kubernetes is a complex tool, but taking your first steps is relatively easy. This is especially true today when all major cloud providers offer easy one-click creation of Kubernetes clusters; you can have a fully working Kubernetes cluster in a matter of minutes. So, what do you do then? You'll probably deploy some pods. Pods are arguably the most important Kubernetes resources. You may have heard about them already, since deploying pods is usually one of the first things in any Kubernetes tutorial. You may have even heard "they're kind of like containers." In this post, you'll learn everything you need to know about pods.

‍

Kubernetes Pods 101

Before Kubernetes, everyone was talking about containers. When you wanted to deploy only one small microservice, you'd say that you needed to deploy "one container." On Kubernetes, everyone talks about pods instead. So, when you only want to deploy one microservice, you'll say that you need to deploy one pod.

Are pods the same as containers, then? Well, not really. A pod is the smallest deployable unit in a Kubernetes world. This means that you can't directly deploy a single container in Kubernetes. If you want one container running, you need to package it into a pod and deploy one pod. A pod can also contain more than one container. It's basically like a box for containers.

Long story short: if you mainly deploy single containers, there isn't much difference between a pod and a container. Technically, a pod encapsulates your container, but in general you can treat it similarly to a container. But pods' ability to contain more than one container is what opens doors of possibilities. We'll dive into that later in this post. But before that, let's talk about pod lifecycles.

Pod Lifecycles

Just like many other resources Kubernetes pods can be in a pending, running, or succeeded/failed state. You can check the status of your pod by executing kubectl describe pod [your_pod_name]:

$ kubectl describe pod nginx-deployment-6595874d85-hnjzw
Name:           nginx-deployment-6595874d85-hnjzw
Namespace:      default
Priority:       0
Node:           k3s-worker3/10.133.106.222
Start Time:     Sun, 21 Aug 2022 12:24:58 +0200
Labels:         app=nginx
                pod-template-hash=6595874d85
Annotations:
Status:         Pending
(...)

As you can see from the snippet above, my pod is in a Pending state. So, what do these states mean?

Pending

Pending, as the name suggests, means that the pod is waiting for something. Usually, it means that Kubernetes is trying to determine where to deploy that pod. So, in normal circumstances, you'll see your pod in the pending state for the first few seconds after creation. But it may also stay in a pending state longer if, for example, all your nodes are full and Kubernetes can't find a suitable node for your pending pod. In such a case, your pod will stay in a pending state until some other pods finish and free up resources or until you add another node to your cluster.

Running

Running is pretty straightforward: It's when everything is working correctly and your pod is active. There is a small caveat to this, though. If your pod consists of multiple containers, then your pod will be in the status "running" if at least one of its primary containers starts successfully. This means there's a chance that your pod will be in a running state even though not all containers are actually running. So, in the case of multiple containers, it's always best to double-check individual container states to be sure.

Succeeded/Failed

Succeeded or failed is what comes after running. As you can imagine, you'll see "succeeded" when your pod did its job and finished as expected, and you'll see "failed" when your pod terminated due to some error. And again, in the case of multiple containers in one pod, you need to be aware that your pod will end up in a failed state if at least one of the containers ends up having issues.

Unknown

The other phase a pod can be in is called "unknown," and you probably won't see it often. A pod will be in a state unknown when Kubernetes literally doesn't know what's happening with the pod. This is usually due to networking issues between the Kubernetes control plane and the node on which the pod suppose to run.

What Are Pods Used for?

Now, the big question: What are pods actually used for? The simple answer would be "to run your application." At the end of the day, the point of running Kubernetes is to run containerized applications on it. And pods are the actual resources that make it possible. They encapsulate your containerized application and allow you to run it on your Kubernetes cluster.

However, it's worth mentioning that usually you won't actually be deploying pods themselves. You'll be using other, higher-level Kubernetes resources like Deployments or DaemonSets that will create pods for you.

Pods vs. Other Resources

Pods are only one of many Kubernetes resource types. Most other types are directly or indirectly related to pods, because as we already said, pods are resources that will actually be running your application on the cluster. Therefore, pretty much anything that your application may need—be it a secret or storage or a load balancer—will all need to somehow relate or connect to a pod.

‍

Kubernetes secrets can be consumed by pods. Kubernetes service resources used to expose a containerized application on your cluster to the network or internet need to reference a pod. Volumes in Kubernetes are mounted to pods. Kubernetes ConfigMaps used to store configuration files are loaded to pods. These are just a few examples, but in general, pods are usually at the center of everything that's happening on Kubernetes.

How to Create A Pod

I'll show you how to create a pod, but be aware that normally you wouldn't create pods directly. You should use higher-level resources like Deployments that will take care of creating pods for you. But if you ever need it for testing or learning purposes, you can create a pod with the following YAML definition:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod-example
spec:
  containers:
    - name: nginx
      image: nginx

You can apply it just like any other Kubernetes YAML definition, using kubectl apply -f:

$ kubectl apply -f pod.yaml
pod/nginx-pod-example created

$ kubectl get pod nginx-pod-example
NAME                READY   STATUS    RESTARTS   AGE
nginx-pod-example   1/1     Running   0          6s

Pods With Multiple Containers

We mentioned pods with multiple containers already, so let's dive into that a bit more. The first thing for you to know is that pods' ability to run multiple containers is not something you should overuse. For example, it's not meant to be used to combine front-end and back-end microservices into one pod. Quite the opposite; you actually shouldn't combine multiple functional microservices into one pod.

‍Why does Kubernetes give you that option then? Well, it's for a different purpose. Putting more than one container into a single pod is useful for adding containers that are like assistants or helpers to your main container. A common example is log gathering containers. Their only job is to read logs from your main container and forward it (usually to some centralized log management solution). Another example is secret management containers. Their job is to securely load secrets from some secret vault and securely pass it to your main container.

‍As you can see, multiple containers in a pod are typically used in the main container + secondary containers configuration. We call these secondary containers "sidecar containers."

‍Of course, even though it's not usually recommended, there's nothing stopping you from combining two containers into one pod. If you have a very specific use case and you think it would make sense in your case, you can add more containers to your pod. You just need to be aware of the consequences of such an approach. The main one is that, in the case of the failure of the pod, both containers will die.

Summary

As you can see, pods are pretty straightforward resources. In most cases, you can treat them the same as containers, but they do offer extra sidecar functionality when necessary.

‍

Learned all you need to for pod basics? Read on to our advanced pod concepts article here!

How to Edit a File in a Docker Container

Regis Wilson — Thu, 08 Sep 2022 17:01:46 +0000

You want to edit a file in your Docker container, but you’ve run into an error that leaves you with none of the tools you need to make your changes. Now what?

Docker intentionally keeps containers as lean as possible with no unnecessary packages installed to maximize performance and stability. Unfortunately, this also means Docker containers don’t have a file editor like Vim or Nano preinstalled.

In this guide, we’ll show you how to install an editor, make the changes you need to, and return the container to its original state, both from the command line and using the Docker extension inside VS Code.

First, though, some housekeeping. It’s considered bad practice to edit Docker files currently running in a production environment, and, once you’ve made your change, you should remove any packages you installed to do so (the editor, for example).

Here’s our step-by-step guide to editing a file in Docker.

Option 1: Edit from the command line

#1 Log in to your container

If your container is not already running, run the container with the following:

docker run --name <yourcontainername> -d -t

To check all your running containers, you can use the command:

docker ps

You should be met with something like this:

‍
This list indicates your target container is up and running. Note that every container has a discrete ID, which we’ll need to gain root access to the container.

To gain root access to the container, run:

docker exec -it <container-id>

You should see something like this:

‍

As you can see, root@<container-id>:/# indicates we now have root access to the container.

#2 Install the editor

It’s a good idea to update your package manager before you install the editor. This ensures that you install the latest stable release of the editor. On Ubuntu, that command is:

apt-get update

To install your preferred editor, such as Vim, Nano or GNU Emacs:

apt-get install <your package manager>

For example, to install Vim:

apt-get install vim

#3 Edit the File

To edit the file, ensure you are in the appropriate directory and use the command:

vim yourfilename.yaml

Once you’ve made the edit to the file, you can remove the editor (in our case, Vim) like this:

apt-get remove vim

Or like this:

apt-get purge vim

The command “remove” will remove only Vim, and no other config files or dependencies involved in the initial install. The command “purge” will remove all config files associated with Vim. In the interest of leaving no trace, the purge command is probably appropriate in this case.

Your package manager may change depending on your OS. These commands are associated with Ubuntu and Vim.

Persisting an editor for regular changes

The above steps are useful for one-off changes, but if you need to make changes often – in a development environment, for example – it’s best to add your editor to your Dockerfile. This will ensure your chosen editor is always available whenever you spin up another instance of your container.

Add your editor to the Dockerfile like this:

RUN["apt-get", "update"]
RUN["apt-get", "install", "vim"]

Every image created with that Dockerfile will have Vim pre-installed and ready to go.

You can replace “Vim” with your editor of choice, such as Nano or GNU Emacs. Keep in mind that the commands in the square brackets are specific to Ubuntu Linux. You may need to adapt these to the operating system you are running in your Docker container.

Option 2: Edit from VS Code

If you prefer to use a GUI editor (for example, if you’d like to use your mouse to navigate through large files, or cut and paste text), you can use VS Code.

This option requires both the Visual Studio Code IDE and the Docker extension from Microsoft. To install the extension, navigate to the extensions tab in VS Code and type in “Docker”.

Be sure to select the Docker extension from Microsoft. This extension allows you to easily manage any containers on your system directly from its UI.

‍

From here, treating a container like any file directory, you can navigate to and open files in that container, and make your changes right in VS Code.

Closing remarks

Now that you know how to edit files in a Docker file, it’s important to take note of the best practice for it.

Editing files in a running Docker container is only recommended when working in a development environment during conceptualisation and when building proof-of-concepts.

Once you’ve made changes to your project in Docker containers, save a new image with those changes in place. This leaves flexibility for testing two containers comparatively while ensuring stability and consistency across containers.

Terraform Kubernetes Deployment: A Detailed Walkthrough

Regis Wilson — Tue, 30 Aug 2022 17:18:55 +0000

Terraform and Kubernetes are two of the most popular tools in their categories. Terraform is widely adopted as the tool of choice for infrastructure as code, and Kubernetes is number one when it comes to orchestrating containers. Is it possible to combine both? Sure! You can use Terraform to deploy your Kubernetes clusters. It's actually quite common, and it lets you deploy Kubernetes just like the rest of your infrastructure. In this post, you'll learn how to do it.

Terraform + Kubernetes: How and Why?

We have two main questions to answer here. How can you deploy Kubernetes with Terraform, and why would you do that? Let's start with the latter.

‍
The answer doesn't differ from "Why would you deploy anything with Terraform?" From that perspective, there's nothing special about Kubernetes, and you get the same benefit by using Terraform to deploy it as with any other infrastructure. You get automation, infrastructure versioning, reliability, and even the ability to perform infrastructure security scanning.

As for how, the answer is actually similar. You can deploy Kubernetes with Terraform just like any other infrastructure. Meaning, you first need to find a Kubernetes resource definition for Terraform (we'll show you that shortly), adjust some parameters for your needs, add it to your Terraform code, and you're done. And just like with any other resource, Terraform will be able to track changes to your cluster and update its configuration after you make changes to the code.

Deploying Kubernetes: First Steps

Enough theory. Let's see how it works in practice. First, you need to find a Terraform provider for your cloud. If you want to deploy Kubernetes on DigitalOcean, you'd need to follow this documentation. For Microsoft Azure, you'd need to head here for details. And for Google Cloud, you need to check here. These are just a few examples. But no matter which cloud provider you're using, the general approach will be the same. For today's example, we'll use DigitalOcean.

To start from nothing, in the simplest scenario, you need to create two files named provider.tf and main.tf. You could do it all in one file, but it's a good practice to separate providers and main resource definitions. In the code below, you can define your DigitalOcean provider for Terraform and pass your DigitalOcean token:

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
}
variable "do_token" {
  default = "[replace_with_your_token]"
}
# Configure the DigitalOcean Provider
provider "digitalocean" {
  token = var.do_token
}

In main.tf you can now define your Kubernetes.

resource "digitalocean_kubernetes_cluster" "test" {
  name   = "test_cluster"
  region = "nyc1"
  version = "1.22.11-do.0"
  node_pool {
    name       = "worker-pool"
    size       = "s-2vcpu-2gb"
    node_count = 3
  }
}

Now that you have your Terraform files prepared, you need three things. First, you need to initiate the DigitalOcean provider. You can do that with terraform init.

# terraform init
Initializing the backend...
Initializing provider plugins...
- Finding digitalocean/digitalocean versions matching "~> 2.0"...
- Installing digitalocean/digitalocean v2.21.0...
- Installed digitalocean/digitalocean v2.21.0 (signed by a HashiCorp partner, key ID F82037E524B9C0E8)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Then you can run your terraform plan, which will show you planned changes to the infrastructure (which in this case should be creating a new Kubernetes cluster).

# terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  + create
Terraform will perform the following actions:
  # digitalocean_kubernetes_cluster.test will be created
  + resource "digitalocean_kubernetes_cluster" "test" {
      + cluster_subnet = (known after apply)
      + created_at     = (known after apply)
      + endpoint       = (known after apply)
      + ha             = false
      + id             = (known after apply)
      + ipv4_address   = (known after apply)
      + kube_config    = (sensitive value)
      + name           = "test-cluster"
      + region         = "nyc1"
      + service_subnet = (known after apply)
      + status         = (known after apply)
      + surge_upgrade  = true
      + updated_at     = (known after apply)
      + urn            = (known after apply)
      + version        = "1.22.11-do.0"
      + vpc_uuid       = (known after apply)
      + maintenance_policy {
          + day        = (known after apply)
          + duration   = (known after apply)
          + start_time = (known after apply)
        }
      + node_pool {
          + actual_node_count = (known after apply)
          + auto_scale        = false
          + id                = (known after apply)
          + name              = "worker-pool"
          + node_count        = 3
          + nodes             = (known after apply)
          + size              = "s-2vcpu-2gb"
        }
    }
Plan: 1 to add, 0 to change, 0 to destroy.

The plan looks good. One resource will be added, and that's your Kubernetes cluster, so you can go ahead and apply the changes with terraform apply.

# terraform apply
(...)
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
  Enter a value: yes
digitalocean_kubernetes_cluster.test: Creating...
digitalocean_kubernetes_cluster.test: Still creating... [10s elapsed]
(...)
digitalocean_kubernetes_cluster.test: Still creating... [7m10s elapsed]
digitalocean_kubernetes_cluster.test: Creation complete after 7m16s [id=49fd0517-a4a5-41e8-997d-1412c081e000]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

If you now head to your DigitalOcean portal to validate, you can indeed see it there.

And that's it! That's how you deploy Kubernetes with Terraform.

‍### Deploying Kubernetes: Next Steps
Now that you know how it works in general, there are a few things that you need to learn next. First, all you've done is deploy basic, minimal Kubernetes. In more realistic scenarios, you'll probably want to parametrize more options for your Kubernetes. This, however, will highly depend on what you actually need. If you know what you need, you can head to the Terraform documentation and check argument reference for your Kubernetes resource. Find what you need and add it to your code.

For example, if you'd like your Kubernetes cluster to automatically upgrade, you can find the following in the documentation:

To make your freshly deployed cluster automatically upgrade, you just need to add the following to your Kubernetes resource definition in main.tf as follows:

resource "digitalocean_kubernetes_cluster" "test" {
  name   = "test-cluster"
  region = "nyc1"
  version = "1.22.11-do.0"
  auto_upgrade = true
  node_pool {
    name       = "worker-pool"
    size       = "s-2vcpu-2gb"
    node_count = 3
  }
}

But you're not there yet. You can quickly see in the DigitalOcean portal that the cluster currently does not automatically upgrade.

Automatic upgrades are disabled now, so you can run terraform plan again to check what Terraform will try to do.

# terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place
Terraform will perform the following actions:
  # digitalocean_kubernetes_cluster.test will be updated in-place
  ~ resource "digitalocean_kubernetes_cluster" "test" {
      ~ auto_upgrade   = false -> true
        id             = "49fd0517-a4a5-41e8-997d-1412c081e000"
        name           = "test-cluster"
        tags           = []
        # (13 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy

As expected, Terraform will now try to update your cluster in place and add an auto-upgrade option to it. Let's go ahead and apply that change.

# terraform apply
(...)
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
  Enter a value: yes
digitalocean_kubernetes_cluster.test: Modifying... [id=49fd0517-a4a5-41e8-997d-1412c081e000]
digitalocean_kubernetes_cluster.test: Modifications complete after 2s [id=49fd0517-a4a5-41e8-997d-1412c081e000]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

The change was quickly applied to your cluster, and if you double check in the portal again, you can see that, indeed, the auto-upgrade option is now enabled.

Destroying Kubernetes

If you no longer want your Kubernetes cluster, you can destroy it just as easily as you deployed it. All you need to do is execute terraform destroy.

# terraform destroy
digitalocean_kubernetes_cluster.test: Refreshing state... [id=49fd0517-a4a5-41e8-997d-1412c081e000]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy
Terraform will perform the following actions:
  # digitalocean_kubernetes_cluster.test will be destroyed
  - resource "digitalocean_kubernetes_cluster" "test" {
      - auto_upgrade   = true -> null
      - cluster_subnet = "10.244.0.0/16" -> null
      - created_at     = "2022-07-24 06:15:34 +0000 UTC" -> null
      - endpoint       = "https://49fd0517-a4a5-41e8-997d-1412c081e000.k8s.ondigitalocean.com" -> null
      - ha             = false -> null
      - id             = "49fd0517-a4a5-41e8-997d-1412c081e000" -> null
      - kube_config    = (sensitive value)
      - name           = "test-cluster" -> null
      - region         = "nyc1" -> null
      - service_subnet = "10.245.0.0/16" -> null
      - status         = "running" -> null
      - surge_upgrade  = true -> null
      - tags           = [] -> null
      - updated_at     = "2022-07-24 06:37:27 +0000 UTC" -> null
      - urn            = "do:kubernetes:49fd0517-a4a5-41e8-997d-1412c081e000" -> null
      - version        = "1.22.11-do.0" -> null
      - vpc_uuid       = "877cc187-97ad-426c-9301-079e3683d351" -> null
      - maintenance_policy {
          - day        = "any" -> null
          - duration   = "4h0m0s" -> null
          - start_time = "10:00" -> null
        }
      - node_pool {
          - actual_node_count = 3 -> null
          - auto_scale        = false -> null
          - id                = "8df9b48c-329d-41f5-899e-b7b896e28e15" -> null
          - labels            = {} -> null
          - max_nodes         = 0 -> null
          - min_nodes         = 0 -> null
          - name              = "worker-pool" -> null
          - node_count        = 3 -> null
          - nodes             = [
              - {
                  - created_at = "2022-07-24 06:15:34 +0000 UTC"
                  - droplet_id = "309670716"
                  - id         = "b82aeb19-78d8-4571-91e6-a0c2cffdb1db"
                  - name       = "worker-pool-c1766"
                  - status     = "running"
                  - updated_at = "2022-07-24 06:19:09 +0000 UTC"
                },
              - {
                  - created_at = "2022-07-24 06:15:34 +0000 UTC"
                  - droplet_id = "309670715"
                  - id         = "6b0d1ecf-4e48-427b-99a9-0e153056238d"
                  - name       = "worker-pool-c176t"
                  - status     = "running"
                  - updated_at = "2022-07-24 06:18:27 +0000 UTC"
                },
              - {
                  - created_at = "2022-07-24 06:15:34 +0000 UTC"
                  - droplet_id = "309670717"
                  - id         = "5ea0e536-96aa-4171-8602-dc0ab19e9888"
                  - name       = "worker-pool-c176l"
                  - status     = "running"
                  - updated_at = "2022-07-24 06:18:27 +0000 UTC"
                },
            ] -> null
          - size              = "s-2vcpu-2gb" -> null
          - tags              = [] -> null
        }
    }
Plan: 0 to add, 0 to change, 1 to destroy.
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.
  Enter a value: yes
digitalocean_kubernetes_cluster.test: Destroying... [id=49fd0517-a4a5-41e8-997d-1412c081e000]
digitalocean_kubernetes_cluster.test: Destruction complete after 1s
Destroy complete! Resources: 1 destroyed.

Just like that, the cluster is gone.

Summary

And there you have it. That's how you can manage Kubernetes clusters with Terraform. You used DigitalOcean Kubernetes for this purpose, but as mentioned before, the process will be exactly the same for other providers. You'll just need to initiate different providers in provider.tf and then adjust the Kubernetes resource definition in main.tf. It's best to follow Terraform documentation for that. You'll find examples and argument references for major cloud providers.

‍

Managing infrastructure with Terraform definitely helps you save time, but did you know that you can also easily spin up an environment on ReleaseHub directly from your docker-compose file? Give it a shot here, and if you want to expand your Terraform knowledge further, take a look at our post about for_each.

‍

6 Docker Compose Best Practices for Dev and Prod

Regis Wilson — Fri, 19 Aug 2022 21:08:06 +0000

Introduction

Docker solves the "but it runs on my machine" problem by introducing containerization. However, with a multifaceted code base, you must simultaneously run several containers like the back and front end. Further, this will require you to leverage tools such as Docker Compose.

Docker Compose is an excellent tool for optimizing the process of creating development, testing, staging, and production environments. With Docker Compose, you'll use a single file to build your environment instead of several files with complex scripts with a lot of branching logic. You can also share this single file with other developers on your team, making it easy to work from the same baseline environment.

This post is about the best practices of Docker Compose for development and production.

What Is Docker Compose Good for?

Docker Compose is a tool for defining and running multi-container Docker applications. It allows you to bring up and link multiple containers into one logical unit. If you want to use Docker containers, you create one container that listens on an unused port on your machine. All other containers will connect to this server container on the same machine. Linking is where the different Docker services are connected and communicate with each other through a central node. This enables them to share data like configuration or databases.

Docker Compose allows you to deploy your application's services as containers and lets you manage these containers as an organized and working whole in a single place—without having to worry about configuring your application's dependencies. For instance, if your app depends on three other services—like a database, an email server, and a messaging server—using Compose means you won't have to manage them individually.

Instead, Docker handles that part for you so that all four services are available within one cohesive environment. This significantly reduces the time needed to get a service up and running. You can make changes simultaneously across all services. Therefore, Docker Compose is an excellent tool for building complex applications that utilize several services.

‍

Docker Compose Best Practices for Development

During development, you may have the advantage of leveraging local storage, which is not the case during production. In production, resources like storage are costly; thus, you must carefully structure the docker-compose file. Essentially, the configuration in development and production slightly differ, and the best practices also differ. Below are the best practices you should employ when using Docker Compose during development.

Mount Your Code as Volume to Avoid Unnecessary Rebuilds

By mounting the project directory (current directory) on the host to code within the container using the new volumes key, you may make changes to the code as you go without having to recompile the image. This also means you do not have to rebuild and push your image to change between development and production environments. You must delete or stop your local Docker machine and start it again.

‍Note: You can also use a link in your code instead of the bind mount, but the downside to this approach is that you'll have to rebuild and repush your Docker image each time you change. With a bind mount, all you need to do is restart the container.

Use an Override File

Some files are only necessary during development and not production. For example, developing a JavaScript application using any of its frameworks needs webpack. Thus, an override file will mimic the compose file but with webpack as a service. When you spin up the container, the compose files will bundle together during development. Therefore, when you make changes in the code base, you'll see the changes in real time. This allows you to have separate settings for the production and development environment by avoiding redundancy. Your docker-compose.override.yml file will have the following:

services:
  webpack:
    build:
      context: "."
      target: "webpack"
    command: "yarn run watch"

Use YAML Anchors

YAML anchors let you reuse parts of your YAML file-like functions. You can use them to share default settings between services. For example, let's say you want to create two services: api and web. Both of them will use Redis as a cache and a database, but the web service will need additional volumes mounted. It would be cumbersome to set all these things in the web service's docker-compose.yml file because it would have to duplicate those lines in the api service's file.

Using YAML anchors to share those settings, you can set the volumes section in the web service's file like this:

x-app: &default-app
  build:
    context: "."
    target: "app"
  depends_on:
    - "postgres"
    - "redis"
  env_file:
    - ".env"
  restart: "${DOCKER_RESTART_POLICY:-unless-stopped}"
  api:
    <<: *default-app
    ports:
      - "8000:8000"
  web:
    <<: *default-app
    ports:
      - "8000:5000"

Additionally, you may alter an aliased property by overriding it in a particular service. If you wanted to, you could set port:5000 in the example above to only the web service. The alias will give it precedence. This method is beneficial when two services could share a Dockerfile and a code base but have some slight variations.

Docker Compose Best Practices for Production

As mentioned, dev and production may have slight configuration differences. So now, let's look at some best practices to help your app be production ready.

Leverage the Docker Restart Policy

Occasionally, you'll face a scenario when a service fails to start. A common reason is that another service on your host machine has changed, and Docker Compose uses the old environment variables. To ensure this doesn't happen, set the restart behavior to restart: always and configure your services with update_config: true. This will refresh the environment variables for each run. However, if your app relies on other services (MySQL, Redis, etc.) outside of Docker Compose, then you should take extra precautions. Make sure they are configured correctly.

Correct Cleanup Order of Docker Images

You need to clean up the order of your images during production. Do not use docker rm -f as it may destroy useful images. Always run docker rm -f --remove-orphans. If you're working in the dev stage, this is not an issue because Docker Compose builds images only once, then exposes them. Thus there's no need to worry about removing old images. However, in production, Docker loops through all images when the container stops and restarts.

Consequently, there's no way for you to be sure that an image wasn't destroyed, even when docker-compose down is called. If a container is stopped and restarted, then the exposed images can change, and you can't be sure they're still in use. Using docker rm -f to delete containers is a mistake. Docker Compose reuses port bindings, so an old service is still available, even though its container was destroyed.

Since you cannot tell which containers might be potentially in use, you must delete all of them using the --remove-orphans flag. If a container is restarted by Docker Compose (or something else) and it reuses the same port, the new image will have the same image ID as the old one.

‍

Notice we've added the --remove-orphans flag because that ensures Docker Compose only deletes containers and images that are no longer in use, regardless of whether we or a running container uses them. This is crucial if you have services restarting.

Setting Your Containers' CPU and Memory Limits

You can configure Docker to limit the CPU and memory of your containers by passing arguments into the docker-compose.yml file before starting your container. For example, the following command will start a web service with one CPU:

web:
    deploy:
      resources:
        limits:
          cpus: "1"

If you set a specific number of CPUs in the multi_cpu key, it will only be used when available. If you fail to set the limit, the service will use the maximum resources it requires.

Tip: If you want to run multiple containers with different memory limits on the same machine, ensure that all your containers have different memory limits. This is because each container views how much memory it needs.

Note: You can use this technique for multiple services if you'd like. Docker Compose will automatically get the values from the env file for each container when it starts up.

Consequently, you need to understand the resource requirements of your service. This will prevent you from wasting resources and minimize production costs.

Conclusion

Hopefully, these tips will help you use Docker Compose more effectively in development and production. After trying out the above configuration and optimization, you should be able to build your containers efficiently. If you feel that the above approach reduces the complexity of your Docker composition setup, don't worry. There are much easier ways to organize your containerized services for development and production at ReleaseHub.

‍

How to Solve AWS EFS “Operation Not Permitted” Errors in EKS

Regis Wilson — Thu, 21 Jul 2022 15:14:00 +0000

At ReleaseHub, we operate dozens of Amazon Elastic Kubernetes Service (EKS) clusters on behalf of our customers. The various workloads and application stacks we have to support are practically as diverse as the number of engineers who use our product. One very common use case is a permanent storage space for the workloads that are deployed in each environment.

The most common general solution for storage in AWS for compute workloads is the Elastic Block Service (EBS), which has the advantage of being relatively performant and easy to set up. However, it has the drawback that EBS volumes are tied to a specific Availability Zone (AZ). Therefore, using Kubernetes workloads running in multiple Availability Zones (AZs), it turns out that ensuring pod workloads correctly map to the correct AZ is actually difficult to do properly and has caused numerous issues for our customers who use EBS storage in their clusters. We also discovered that EBS storage costs can add up quickly and over-provisioning volume sizes (which is a necessary evil) can add to this problem.

Without going too far down the pros and cons of each storage system, we found that most customers were well satisfied with using Elastic FileSystem (EFS) mount points to provide persistent storage volumes backing the application workloads deployed to their clusters. EFS provides a good balance of performance, reliability, price (pay-for-what-you-store), and AZ diversification. As such, we made an early decision to move almost all customer workloads off EBS to EFS and only allowed the EBS option for customer workloads who specifically opt-in to it. This solution worked well for us since EKS version 1.14 all the way up until recently when we started moving customers to 1.21 and beyond.

The Problem

In our original implementation of EFS workloads in EKS, we started out using the (now retired) EFS provisioner. This solution allowed our customers to specify a volume for persistent storage and the provisioner would create a filesystem mount from an existing EFS infrastructure point (which we create automatically upon cluster creation). The customer pods would then mount this filesystem and have unlimited storage that would persist until the workload expired or was deleted, at which point the volume space would be removed. We literally experienced zero issues with this configuration from the first time we tested it.

In recent months, we have been tirelessly upgrading to the latest version(s) of EKS to keep customers up to date with the latest features and deprecations in the never ending Kubernetes versions. Upon reviewing the various addons and plugins, we realised that the EFS provisioner was replaced by the modern EFS CSI driver. You can read more about the two projects in this stack overflow article.

The upgrade process was not terribly difficult for us since we could easily run both provisioners side by side and then switch over workloads using the Kubernetes Storage Class objects. As one example, Customer A would be using the legacy provisioner: releasehub.com/aws-efs storage class and then we could upgrade any subsequent workloads to provisioner: efs.csi.aws.com and then test until we were satisfied with the results. Rolling back was easy to revert the workloads back to the original storage class.

Eventually, after demonstrating that the process worked seamlessly and nearly flawlessly with the new driver and the same infrastructure in a variety of scenarios, we were able to confidently roll out the changes to more and more customers in a planned migration.

That was when we ran into two major stumbling blocks with customer workloads that use persistent volumes: postgres and rabbitmq containers. Here are the horrible details we discovered for each:

initdb: could not change permissions of directory "/var/lib/postgresql/data/pgdata": Operation not permitted

‍chown: /var/lib/rabbitmq: Operation not permitted

It is important to note that this could happen to any workloads that use the chown command, but these were the most common complaints we got from customers.

Diagnosis

At first, we did what every engineer does: we searched Google and confirmed the problems were widespread, finding stack overflow and server fault questions here and here respectively. Unfortunately, and most frustratingly, there were no good solutions to the problem(s) and even worse, many of the solutions posited by people were highly complex, tightly tied to a particular implementation, or technically brittle. There seemed to be no particularly elegant, easy solution especially for our wide diversity of customer user cases.

We tried using the latest versions of the drivers to no avail. We tried even older versions of the CSI driver to see if this might have been a regression (to no avail). Digging in even deeper to EKS and EFS specifically, we discovered that dynamic provisioning (which is what we rely on to provide a seamless, fast, efficient service for workloads) was recently added to the new CSI driver. This GitHub issue (unsolved to this day) indicates that the problem has actually been in place from the beginning of the driver’s use cases.

Reading through the various use cases affected was like reading a long-lost diary of all our horrible secrets and failures laid bare: including some horrific harbingers of doom we had nearly inflicted on the rest of our customers who were yet to be migrated. We quickly reviewed our test cases and made the stunning discovery that we had been testing all kinds of workloads that read and write to NFS volumes, but hadn’t tested the ones that use chown. That was the only use case we hadn’t considered, and it was the one use case that failed.

The root cause of the issue is that an EFS mount point that is dynamically created for a pod workload is given a set of mapped numerical User IDs (UIDs), but the UID that is stored inside the pod workload typically will not match the UID assigned to the EFS mount point. In most use cases, the operating system will not necessarily care what UID is in use on the mounted filesystem; it will typically just blindly read and/or write to the filesystem and assume that if the operation is a success that the permissions are correct. There are a number of good reasons not to be that trusting however. For example, in a database scenario, the permissions related to reading and writing data for the storage of important information is not left to chance and the application will attempt to ensure the UID (and maybe even Group IDs [GIDs]) match.

This did not answer the question of why the legacy deprecated provisioner seems to work flawlessly, but we will dig into that on another blog post.

To date, there does not seem to be any way to match the UIDs so that the operating system inside the container can set or even pretend to set the UID of a directory the application needs for reading and writing so that it matches the physical infrastructure underlying Kubernetes. This is not just an academic legacy issue, it is a real concern for security and privacy reasons that affect modern applications running in modern Cloud Native environments.

A Few Solutions

Finally we present a few solutions, in chronological order of ones that we tried. We gradually settled on the last option as you will see the rationale behind this decision unfold.

Option 1: Find every occurrence of Waldo and fix it for each customer and application workload. This option sounds as bad as you imagine it would be. Worse, it could make an easy and simple solution (pull a standard container and run it) unusable under normal circumstances. Even worse, our work would never be done: any new customers we onboard would have a new set of changes or fixes or workarounds to find and implement.

For example, we could easily identify the lines affecting us in the postgresql image entrypoint and create our own version. Which you would then need to create a separate dockerfile and modify it to your tastes…for each customer and each version of postgres and operating system that is in use times the number of applications each customer uses. Or, we could try to force the UID and GID numbers to match the CSI provisioner’s UID and GID to match (again, with a splinter version of the dockerfile). Now that we have quote-unquote, allegedly, supposedly, air quotes “solved” the problem, do the exact same thing for the next application (like rabbitmq, or Jenkins, or whatever) and all the application and operating system versions. Not just now, but also moving forward into the future forever.

Option 2: Try to boil the ocean to find every single species of fish and identify them. Taking a step back, it is clear that we cannot hope to ever solve every use case of chown that is out there in the wild today, not to mention new ones that are being born every year. We were able to identify that most docker images use a specific UID and GID combination and the numbers of these are fairly limited. Examining two use cases in question, we found that postgresql images tended to use 999:999 and several others used 99 or 100, perhaps 1000 and 1001. This seemed like a promising lead to a solution because you can specify the UID in the CSI provisioner.

This elegant solution would result in creating several StorageClasses in Kubernetes, like say, “postgresql-999”, “rabbitmq-1001”, and so forth. Or maybe just “efs-uid-999” to be more generic. Then we would teach each customer who enjoyed a failed build or deploy stack trace to change their settings to use the appropriate StorageClass. Even better, there are only about 2^16 possible unique UIDs in Linux, so we could programmatically create all of them in advance and apply them to our cluster to be stored in etcd, ready for retrieval whenever a customer wanted a UID-specific storage class. Or to limit choices in an opinionated but friendly way, we could require all containers to use a fixed UID, like 42, in order to use the storage volumes on our platform. If a customer wanted to use a different UID, like 43, we could charge $1 for every UID above and beyond the original one.

If you did not detect any sarcasm in the preceding paragraph, you may want to call a crisis hotline to discuss obtaining a sense of humour. Amazon does not sell any upon last check; although you might find a used version on Etsy or eBay. I once ordered a sense of humour and it was stolen by a porch pirate before I could bring it in. Once I had obtained a suitable one, I would occasionally rent mine out on the joke version of Uber or Lyft, and sometimes you can even spend the night in my sense of humour on AirBNB, but due to abuse and lack of adequate tipping I have had to scale my activities down lately.

Option 3: When in doubt, rollback to when it worked. We ultimately had to decide that we would be unable to support the new CSI driver until an adequate solution for dynamic deployments of EFS volumes was found for EKS. In the world of open source, there is always someone who comes up with a clever solution to a common problem and that becomes the de facto implementation recommendation. Currently, we were satisfied with the original functionality of the deprecated provisioner.

But this raises another issue, how do we square using a deprecated and potentially unsupported solution on a platform our customers depend and rely upon? The answer is that we can make small adjustments and updates to the yaml and source code since the original solution code is still available and can be updated by Releasehub to support our customers.

Conclusion

Sometimes we must accept that we live in an imperfect world and accept the fact that we are as imperfect as the imperfect world we live in which means that we should accept the imperfection as the correct way that things should be and thus, the imperfection we see in the world merely reflects the imperfections in ourselves, which makes us perfect in every way.

How To Write Route53 Stubbed Responses For Rspec Tests

Regis Wilson — Tue, 14 Sep 2021 16:05:32 +0000

In this blog post, I will go over a recent exercise to fix some bugs, refactor, and write tests for some of our code related to Route53. Route53 is an AWS service that creates, updates, and provides Domain Name Service (DNS) for the internet. The reason that code unit tests are so important is because it helps reveal bugs, creates supportable and high quality code, and allows restructuring and refactoring with confidence. The downside to writing unit tests is that it can be time consuming, difficult at times, and bloating to the normal code base. It is not uncommon for unit tests’ "lines of code" (LOC) count to far exceed the LOC for the actual codebase. You would not be crazy to have nearly an order of magnitude difference in LOC for actual codebase versus LOC for unit test cases.

In this case, interacting with the AWS Route53 API was daunting to test and stubbing responses seemed incredibly difficult until I found some examples written by another one of our engineers that showed how the rspec and API SDKs could be made to work in a fairly straightforward and (dare I say) downright fun method for unit testing Ruby code.

The Code Under Examination

This straightforward code snippet was my first target for unit testing. It is very simple and only does one thing. It is ripe for refactoring for readability and reusability for other sections of the code. This should be the best way to begin the project and get familiar with the rspec templates I’d be using later. Before I start refactoring and fixing bugs, I wanted to write tests. Other than the fairly “inliney” and hard to follow syntax and “magical” code, can you spot any bugs?

def route53_hosted_zone_id(subdomain)
  route53.list_hosted_zones_by_name.map do |response|
    response.hosted_zones.detect{|zone| zone.name == "#{subdomain}." }&.id&.gsub(/.*\//, '')
  end.flatten.compact.first
end

Write Helpers Before the Refactor

I am already itching to remove the magical subdomain rewriting and gsub deleting into separate methods that can be reused and are easier to read:

def cannonicalise(hostname)
  hostname = domain_parts(hostname).join('.')

  "#{hostname}."
end

def parse_hosted_zone_id(hosted_zone_id)
  return nil if hosted_zone_id.blank?

  hosted_zone_id.gsub(%r{.*/+}, '')
end

Stub and Test the New Methods

First things first, we need to do a little bit of boilerplate to get the API calls mocked and stubbed, then add a few very simple tests to get started.

# frozen_string_literal: true

require 'rails_helper'

RSpec.describe Cloud::Aws::Route53 do
  let(:route53) { Aws::Route53::Client.new(stub_responses: true) }

  subject { FactoryBot.create(:v2_cloud_integration) }

  before do
    allow(subject).to receive(:route53).and_return(route53)
  end

  describe '#parse_hosted_zone_id' do
    context 'with a valid hostedzone identifier' do
      it 'returns just the zoneid' do
        expect(subject.parse_hosted_zone_id('/hostedzone/Z1234ABC')).to eq('Z1234ABC')
      end
    end
  end
  describe '#cannonicalise' do
    context 'without a dot' do
      it 'returns the zone with a dot' do
        expect(subject.cannonicalise('some.host')).to eq('some.host.')
      end
    end
    context 'with a dot' do
      it 'returns the zone with a dot' do
        expect(subject.cannonicalise('some.host.')).to eq('some.host.')
      end
    end
  end
end

Write A Fixture

Perfect, now we can test our new cannonicalise and parse_hosted_zone_id methods and we have a stubbed response coming from the Route53 API calls. Let’s write a simple new test to uncover some bugs by testing the api responses we get. The first step is to write some fixtures we can test with. Here we generate two faked stubbed responses for a very common domain.

context 'an AWS cloud integration' do
    before do
      route53.stub_responses(:list_hosted_zones_by_name, {
                               is_truncated: false,
                               max_items: 100,
                               hosted_zones: [
                                 {
                                   id: '/hostedzone/Z321EXAMPLE',
                                   name: 'example.com.',
                                   config: {
                                     comment: 'Some comment 1',
                                     private_zone: true
                                   },
                                   caller_reference: SecureRandom.hex
                                 },
                                 {
                                   id: '/hostedzone/Z123EXAMPLE',
                                   name: 'example.com.',
                                   config: {
                                     comment: 'Some comment 2',
                                     private_zone: false
                                   },
                                   caller_reference: SecureRandom.hex
                                 }
                               ]
                             })
    end
end

If you’re wondering how to make these fixtures, you can easily read the AWS Ruby SDK V3 documentation for sample inputs and outputs, or you can make API calls via the AWS CLI and inspect the responses, or you can even just put in some values and see what happens when you run rspec. For example, if I remove, say, the caller_reference parameter, I’ll get an error that helpfully identifies the problem.

You really can’t go wrong with the SDK validation and stubbed responses taken from the examples or from live requests you make with the CLI! This is already a tremendous benefit and we’re not even testing our own code yet.

Write a Test Case with the Stubbed Responses

Now we can write some unit test cases and loop through several responses that we expect to find the hosted zone. Voilá we’ve uncovered some bugs just by being a little creative with our inputs! Do you see why?

describe '#route53_hosted_zone_id' do
  %w[
    example.com
    example.com.
    www.example.com
    www.example.com.
    test.www.example.com
    test.www.example.com.
    deep.test.www.example.com
  ].each do |hostname|
    context 'for hosts that exist in the parent zone' do
      it "returns the hosted_zone_id for #{hostname}" do
        expect(route53).to receive(:list_hosted_zones_by_name).with(no_args).and_call_original
        hosted_zone_id = subject.route53_hosted_zone_id(hostname)
        expect(hosted_zone_id).to eq('Z123EXAMPLE')
      end
    end
  end
end

What these failed test cases are telling us is that the code worked under perfect conditions but in strange scenarios that may not be uncommon (for example, having an internal private zone and public zone with the same name, or selecting a two-level-deep name in a zone) could cause unpredictable behaviours.

The Solution is an Exercise for the Reader

Now we merely need to write or refactor the code from our original snippet to pass all of our new test cases. One of the issues that our test cases revealed was that two-level-deep names (say, test.www.example.com in the zone example.com) would be missed. We also needed a way to ensure that zones are not private, perhaps with an optional parameter to specify private zones. Here is an example that passes all the existing tests and welcome feedback on any other bugs or optimisations you find.

def route53_hosted_zone_ids_by_name(is_private_zone: false)
  # TODO: danger, does not handle duplicate zone names!!!
  hosted_zone_ids_by_name = {}
  route53.list_hosted_zones_by_name.each do |response|
    response.hosted_zones.each do |zone|
      if !!zone.config.private_zone == is_private_zone
        hosted_zone_ids_by_name[zone.name] = parse_hosted_zone_id(zone.id)
      end
    end
  end
  hosted_zone_ids_by_name
end

def route53_hosted_zone_id(hostname)
  # Recursively look for the zone id of the nearest parent (host, subdomain, or apex)
  hosted_zone_ids_by_name = route53_hosted_zone_ids_by_name

  loop do
    hostname = cannonicalise(hostname)
    break if hosted_zone_ids_by_name[hostname].present?

    # Strip off one level and try again
    hostname = domain_parts(hostname).drop(1).join('.')
    break if hostname.blank?
  end
  hosted_zone_ids_by_name[hostname]
end

Congratulations

All test cases now pass! Keep writing tests until you get nearly 100% coverage!

Kubernetes Health Checks - 2 Ways to Improve Stability in Your Production Applications

Regis Wilson — Thu, 25 Mar 2021 00:11:26 +0000

Kubernetes Health Checks - 2 Ways to Improve Stability in Your Production Applications

Health checks are often a last-minute addition to your application stack, if they are even included at all. Advanced Site Reliability Engineering (SRE) practices try to push best practices (like health checks) forward so they are included early before applications are deployed. Many engineers know intuitively that health checks are important, but getting them implemented correctly—and keeping them up to date—is very hard. This article tries to document best practices for health checks, application development including SRE tenets, and how to improve the stability and even performance of your application when it runs in production.

At Release, we have previously written about how we monitor and configure applications so you can also read that blog post now or at a later time.

Costs of Downtime and Instability in Production

Do you know how much it costs for your application to go offline? Don’t worry if you don’t—or can’t—know the exact figure: the important thing is to go through the mental process of estimating how much an outage or degradation to your application would “cost.” Costs are not only measured in currency, you need to also consider impacts to your brand, your Net Promoter Score (NPS), chatter online and on social media among customers and potential customers, and even negative reactions in the public media.

I have worked in Site Reliability and DevOps my whole career and I have worked at many different companies whose responses for downtime ranged from the casual “our site will be back up eventually and we’ll be fine” to “we have lost $XXX per minute in revenue and we need to investigate methods for replacing that revenue”. No matter the response, I still did the best job my team and I could muster to keep the application and infrastructure services alive and well. There will always be bugs and issues with the code that is deployed and how it runs, however, if a problem occurs at a lower level in the application stack or in the infrastructure itself, then the application simply has no hope of servicing the needs of the consumers who visit your site.

The metaphor that I used often was one of cars driving on the motorway: if the roads are wet and slippery, then the cars will be unsafe and dangerous. When and if a crash occurs, then the roads will also be blocked and traffic will stop while the crash is cleaned up. It’s true that the cars may run out of gasoline, the drivers may get lost and go to the wrong destination, or the cars may not have good horsepower to drive quickly, but all of those factors are a higher order concern in the traffic stack. In this way, I saw my team’s and my job as keeping the roads as clear and safe and uncongested as possible so that the cars could operate at the best possible level.

The Symptoms Are The Disease

Very early in the internet industry, the best practices for application stability were primitive and reactionary. Site reliability involved a manual post-mortem approach: finding out what happened and then applying monitoring and alerting on that behavior to alert an operator that something was wrong. The best practices at the time involved a team of on-call engineers and operators who would literally watch an application 24 hours a day, 365 days a year (one extra day for leap years) and respond within a certain timeframe (usually less than fifteen minutes) to manually investigate and fix any issues that came up. In some cases the “team” was actually one poor person tasked with the impossible job of being on-call indefinitely.

There are several drawbacks to this approach, not the least of which is the human toll such manual response takes and the unsustainable pace. The cost of the team, the cost of staff turnover and training, the losses due to turnaround time and missed calls, and the impact to end users were all huge reasons to implement a better solution.

Health Checks to the Rescue

One key initiative that came about in the early aughts was the concept of a health check in the load balancer. I was part of a team that worked with several major load balancer manufacturers to implement a way to not only route traffic to services in our application, but to add monitors and tests (even then we called them “health checks”) to the endpoints which would allow us to add or remove services that were not responding or were unhealthy. The concept was that a web application would respond on a well-known port and respond with a well-known response that proved the application was ready to serve traffic.

For example, we might query the backend service at http://192.168.0.10/health-check and we expected the service to respond with a string like 200 OK. This trivial example doesn’t sound like much until you realized that our end-goal was to actually perform some internal checks in the application which would allow us to do more than respond with a static string. For example, the application might check that the database is responding to a sample query that a table exists, and then the application could check that the CPU was at some nominal value. Therefore, the health check could be expanded to something like:

HTTP/1.0 200 OK

Checking DB… ok
Checking CPU… ok
Checking User cache… ok

Conversely, if something went wrong, the application could respond something like this:

HTTP/1.0 500 CRITICAL

Checking DB... ok
Checking CPU… ok
Checking User cache… CORRUPTED

Using the response code of 200 and looking for the string “OK” (for example), the load balancer manufacturers were able to remove a service from the backend pool, allowing other servers to accept requests and avoid servers that would otherwise have an error. Also, we could set a timeout so that the load balancer would consider no response to be an error. In this way, we can remove traffic from servers that were not responsive. The beauty of the system we were designing was that we were going to be able to monitor errors proactively and directly at the origin. The servers would be removed before they became a problem.

We would also use the same health check in our monitoring and alerting systems that we had perfected over the previous decades by manually watching them and using them for diagnosis. The difference is that we had more information about what was going wrong, and simultaneously we had more time to respond and properly diagnose the problems without affecting customers at all. Imagine the relief at not having to respond to every alert at 2AM within 15 minutes, but being able to automatically open a ticket to have a technician during the graveyard shift respond within the hour and restart the server and add it back to the pool.

Even better, we were able to convince the load balancer manufacturers to implement an inline-retry policy based on the same idea. For example, if a live service request to a backend server failed with a 500 error code, the load balancer could not only remove the server from the pool, but it could retry the request one more time on a healthy server. With this technology, the loadbalancer could try to resolve the situation before the customer even noticed anything was wrong, and no human could be quicker.

It’s Not Just Human Labor

We did better than save human labor and effort in monitoring the systems and responding to problems. After implementing health checks on the load balancers and inside the application, we were able to reduce errors and outages to the point where we actually raised our traffic levels by a double-digit percentage, and also increased actual revenue by a measurable amount. Users who might have encountered an error and navigated away after a Google search were staying around to browse and (more importantly) make purchases. By further tweaking the load balancing algorithms to favor healthier (or faster) servers, we further increased this beneficial business result even further. Steady growth over time occurred as well, because Google saw improved signals from users and fewer errors and therefore moved the site up in rankings. This was a stunning and unexpected outcome that was attributed to removing errors and downtime from our application running with this infrastructure and by utilising SRE practices (long before the term was coined).

The Modern Solutions

With the advent of Kubernetes, the lessons learnt the hard way over the past few decades have been carried forward in architecting a resilient and reliable design for complex service interactions. Kubernetes uses the concepts of a probe to test the application for liveness and readiness (there is a third probe that tests for startup delay, but we’re skipping that for the purposes of this article). With these two probes, we can implement a solution that makes applications far more stable and reduces downtime and manual intervention.

Liveness Probes

The first solution is the liveness probe which has the job of figuring out if a service is responding properly and within a certain time frame so that it can be considered running properly. If the probes fail, then the pod is considered “dead” and the pod will be terminated and restarted somewhere else in the cluster. For example, a web server may have a memory leak and stop responding after a certain amount of time or number of requests have occurred. Another example might be a database that fills up a temporary disk space area and is unable to process further transactions until the space is cleared out.

You may be saying to yourself that these seem like errors that should be corrected and dealt with properly rather than simply killing the pod and waiting for it to be rescheduled somewhere else. You would be absolutely correct, but let me counter with a rhetorical question asking, “Given this error condition, what do you want me to do at 2AM when no one is available?” The liveness probes can be excellent at monitoring non-responsive servers without state, but may not be so great at monitoring and restarting services with state, like the database example I gave above. So we recommend using the liveness probe only if you feel it would help more than it would hurt. We also spend extra care and effort to ensure that the liveness probes are very forgiving so they do not trigger on false-positive alarms.

Another counter-argument to the “fix it” stance has to do with direct or indirect engineering costs and interacting with third party or open-source code. Trying to allocate resources to fully diagnose an intermittent problem, much less attempt to fix the problem can be difficult. In the case of a third party software or an open-source project where getting upstream fixes submitted, prioritized, approved, tested, and pulled back downstream can be enormously expensive and time consuming. Sometimes the answer really is “just restart it”.

Liveness Probes

The second solution is a readiness probe which is much like the solution I described earlier in this article with the load balancers. Indeed, the readiness probe does exactly what I’ve described: Kubernetes will periodically run a command to test the service running inside the container to gauge proper and timely responses. The ingress (just a fancy name for the load balancer) will not send traffic to this pod unless and until the readiness probe states that the service is ready for correct operation.

This helps in some scenarios where a web server may hit a threshold in connections or traffic levels where it may slow down or stop responding to new requests. It may be the case that the application simply cannot handle more than a certain number of transactions and so Kubernetes can use this signal to route traffic to another pod that is less busy. If this slow down or refusal to respond can be correlated with other metrics (like traffic volume, CPU utilization, etc.) then the horizontal pod autoscaler could trigger more resources to be added to the service.

In fact, we believe that readiness probes are so important to correct functioning of applications that we strongly recommend all services have a health check of some kind enabled and tested. We feel so strongly about this that we have considered making it a warning condition when no health check is configured on a running service in any of your environments at Release. Specifically, we could make readiness probes an opt out requirement rather than an opt in nicety.

Some examples

Here are some actual examples of health checks that we have implemented for our customers. These examples are generic enough to be applied almost anywhere.

In this example we do a simple Nginx check on port 80 to ensure that the application is responding before we send traffic to the proxy.

 readiness_probe:
   exec:
     command:
     - psql
     - "-h"
     - localhost
     - "-c"
     - SELECT 1
   period_seconds: 2
   timeout_seconds: 2
   failure_threshold: 30

In this example we perform a health check against an Elastic search node to ensure that the cluster is healthy before accepting traffic (which presumably cannot be processed yet). This is a straight port from the Docker Compose examples in the open source repositories.

- name: elasticsearch
 image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
 ports:
 - type: node_port
   target_port: '9200'
   port: '9200'
 liveness_probe:
   exec:
     command:
     - curl
     - "--fail"
     - localhost:9200/_cluster/health
   timeout_seconds: 2
   failure_threshold: 3
   period_seconds: 30

This example is good to show how a non-HTTP check for a postgres database can be used to ensure the database is up and responding to requests. Note that if this database is not clustered, then application database requests can fail when the health check fails. Your application will need to respond accordingly (either fail in turn to cascade a failover at a higher level, or perform some sort of mitigation so that a graceful failure happens). Recall that if this were a liveness probe, the postgres container would be killed and restarted, which may not be what you want at all.

 readiness_probe:
   exec:
     command:
     - psql
     - "-h"
     - localhost
     - "-c"
     - SELECT 1
   period_seconds: 2
   timeout_seconds: 2
   failure_threshold: 30

Step N, Profit

By implementing either (or both!) of these health checks, you can not only reduce the amount of time humans have to spend monitoring and interfering with applications, but you can even dramatically improve your traffic response levels, response times, and performance. In some cases, you might even be able to measure the impact to your customers’ NPS and/or your company’s top and bottom line.
Photo by Hush Naidoo on Unsplash