DEV Community: Shan Desai

Exploring the Marshland of Docker Compose Versions with single quotes, special characters and environment variables

Shan Desai — Fri, 15 Jul 2022 16:06:41 +0000

Problem

Since Docker's Compose V2 General Availability announcement, a lot of things have changed which might lead to
Incompatibility in your currently running application stacks.

Compose V2: what changes?

instead of docker-compose the syntax will be docker compose
docker-compose (v1) was written in Python, docker compose (v2) is in Go
Compose V1 is official EOL (End Of Life) aka. Deprecated

There are lot of fine things in Compose V2 but this post isn't about it. This write-up takes you through the murky
marshland between Compose V1 and Compose V2 and how things might start you break when you decide to jump to the
cool V2 vers without doing some thorough checks in your stack application

Swiss Knife for the Exploration

Tools you will need:

Hashicorp Vagrant
Ansible
Different versions of Docker Compose (v1 as well as v2)

Vagrant

Instead of setting up a pain-staking Virtual Machine, we will leverage a bit simplified and inexpensive Vagrant Machine
with Ubuntu 20.04 (focal amd64) base image.

Within this image we will install the following:

Docker Engine 20.10.17
Docker Compose v1: v1.25.4 build 8d51620a and v1.29.2 build 5b3cea4c
Docker Compose v2: 2.6.0

Ansible

I have Ansible installed on my host machine

ansible --version
ansible [core 2.13.1]

Ansible will install all the spicy Compose versions into the Vagrant Machine for us

Why are we doing this?

If you have containers that require passwords at rest e.g. in .env environment files to be encrypted and with
special characters like the $ in them, we are going for ride which will make you want to bookmark this write-up.

It turns out, I had a system running Compose v1 (v1.25.4) and upon upgrading to Compose v2 (2.6.0) I wasn't able
to login to my Node-RED container which had authentication setup already.

Node-RED Container

Node-RED requires that the administrator password must be encrypted in bcrypt type scheme and the password looks
something like your dog walked over your keyboard.

$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.

This looks fairly okay, but the menace is the $ characters in them.

Let's dive deep shall we!

Code

`Vagrantfile`

I am using a Linux Machine with Virtual Box as a provider

# -*- mode: ruby -*-
# vi: set ft=ruby :


Vagrant.configure("2") do |config|

  # Use Ubuntu 20.04 as base
  config.vm.box = "ubuntu/focal64"

  config.vm.box_check_update =false

  # Synchronize all the docker-compose related file to `/vagrant` dir in VM
  config.vm.synced_folder ".", "/vagrant"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
  end

  # Setup the VM with all the different variants of Docker Compose in them
  config.vm.provision "ansible" do |a|
    a.verbose = "v"
    a.playbook = "./docker-compose-playbook.yml"
  end
end

Ansible Playbook: `docker-compose-playbook.yml`

We will let Ansible install all the required Docker Compose versions for us in the VM through the playbook.

Docker Compose v1.25.4 will be available in the VM as CLI docker-compose-125
Docker Compose v1.29.2 will be available in the VM as CLI docker-compose-129
Docker Compose v2.6.0 will be available in the VM as CLI docker compose

---
- hosts: all
  become: yes
  tasks:
    - name: install prerequisites
      apt:
        name:
          - apt-transport-https
          - ca-certificates
          - curl
          - gnupg-agent
          - software-properties-common
        update_cache: yes
    - name: add apt-key
      apt_key:
        url: https://download.docker.com/linux/ubuntu/gpg
    - name: add docker repo
      apt_repository:
        repo: deb https://download.docker.com/linux/ubuntu focal stable
    - name: install docker
      apt:
        name:
          - docker-ce
          - docker-ce-cli
          - containerd.io
          - docker-compose-plugin
        update_cache: yes
    - name: add userpermissions
      shell: "usermod -aG docker vagrant"
    - name: Install docker-compose v1.29.2 from official github repo
      get_url:
        url : https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64
        dest: /usr/local/bin/docker-compose-129
        mode: 'u+x,g+x'
    - name: change compose v1.29.2 user permissions
      shell: "chmod +x /usr/local/bin/docker-compose-129"
    - name: Install docker-compose v1.25.4 from official github repo
      get_url:
        url: https://github.com/docker/compose/releases/download/1.25.4/docker-compose-Linux-x86_64
        dest: /usr/local/bin/docker-compose-125
        mode: 'u+x,g+x'
    - name: change compose v1.25.4 user permissions
      shell: "chmod +x /usr/local/bin/docker-compose-125"

Files for reproduction of incompatibility

We wont be spinning to much complex stuff so a simple compose file and the relevant .env.node-red
file with the encrypted password should suffice

`docker-compose.yml`

version: '3.7'
services:
  node-red:
    image: nodered/node-red:2.2.2
    container_name: test-nodered
    env_file:
      - .env.node-red

`.env.node-red`

# Encrypting plain-text password with value `password` with bcyrpt
NODERED_ADMIN_PASSWORD=$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.

Compatibility Checks

Let's bring the Vagrant Machine up using:

vagrant up

this will download the base ubuntu image if you previously don't have it on your host machine,
provision it with all the docker compose versions in it.

vagrant ssh

Once into the Machine head to the /vagrant directory, and just to be safe we check if all the
required compose versions are available or not.

$ cd /vagrant

$ docker-compose-125 --version
$ docker-compose-129 --version
$ docker compose version

Checks: Password not encloses in Single Quotes

As the .env.node-red file currently shows that my encrypted password is not enclosed in single quotes

Let's understand what Compose thinks of the value of the environment variable.

Compose v1.25.4 (without single quotes)

docker-compose-125 config

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
version: '3.7'

NO Problem! My dollar characters are escaped by docker-compose. I can login with password password

Compose v1.29.2 (without single quotes)

docker-compose-129 config

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
version: '3.7'

NO Problem! My dollar characters are escaped by docker-compose. I can login with password password

Compose v2 without single quotes)

docker compose config

produces

name: node-red-app
services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: a$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
    networks:
      default: null
networks:
  default:
    name: node-red-app_default

Oh NO! What happened here? Password looks eaten up. The YAML file looks different too!

Checks: Password enclosed in single quotes

Now let's enclose our encrypted password between single quotes, so our .env.node-red file
should look like:

NODERED_ADMIN_PASSWORD=$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.

Compose v1.25.4 (with single quotes)

docker-compose-125 config

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: '''$$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.'''
    image: nodered/node-red:2.2.2
version: '3.7'

Oh NO! Why are there more single quotes than expected. This is not going to let me login!

Compose v1.29.2 (with single quotes)

docker-compose-129 config

produces

services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
version: '3.7'

NO problem! here this will work fine as the dollar characters are escaped safely by compose

Compose v2 (with single quotes)

docker compose config

produces

name: node-red-app
services:
  node-red:
    container_name: test-nodered
    environment:
      NODERED_ADMIN_PASSWORD: $$2a$$08$$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.
    image: nodered/node-red:2.2.2
    networks:
      default: null
networks:
  default:
    name: node-red-app_default

NO Problem! Phew ! This will work just fine !

Result: Compose Compatibility Matrix

Compose Version / Values	`1.25.4`	`1.29.2`	`2.6.0`
with Single Quotes	❌	✅	✅
without Single Quotes	✅	✅	❌

What did we learn today, kids?

No Single Quotes:

Docker Compose v1.25.4 was able to resolve special characters without having enclose them in single quotes
Docker Compose v1.29.2 will also work just fine for values not enclosed in single quotes
Docker Compose v2 will try go hay-wire (try to resolve the value after the $ sign) if you do not use things in single quotes

With Single Quotes:

Docker Compose v1.25.4 will literally take things enclosed in single quotes as part of your environment variable and will
break your Container's Login logic
Docker Compose v1.29.2 will work well
Docker Compose v2 will work well

In a nutshell

DON'T JUMP THE GUN, when moving from different V1 to V2 in Compose. Do some throrough research as well as experiments.

Since Compose V1 is now deprecated, when transitioning to Compose V2 check all your environment variable files and if they require
special characters to be either enclosed in single quotes or not. Estimate your downtimes and conduct isolated tests when required.

Rule of thumb from Compose V2 onwards: if it got some special char in them password, pack em between single quotes

but to each their own!

Github Gist

GitHub Gist available publicly

Multi-Image Docker Images: Using COPY with Images directly from registries

Shan Desai — Fri, 24 Jun 2022 20:10:18 +0000

Problem

I was on the lookout for a solution to a problem in Docker that needed to do the following:

Create a Docker Image for a bunch of compiled ASP.NET .dll files and try reducing the
container footprint

Sounds simple, right? But it turned out to be a bit more messy than I imagined. To top it off,
I have NEVER even programmed in .NET so I was feeling like a visually impaired person made to
navigate his way through a dark room with furniture scattered all over!

Thousand Leagues under the Container Sea!

Keeping the whole Maritime theme alive with Docker (and Kubernetes), I jumped into the sea of
Docker Hub with millions of containers and found out that Microsoft hosts all the .NET related
container images as .NET by Microsoft registry.

Gentle reminder that I have never even thought of .NET before in my life, so just trying to figure out
whether I need an one or more of the following images:

SDK
ASP.NET Core Runtime
.NET Runtime
.NET Runtime Dependencies
.NET Monitor Tool
.NET Samples

was a side-mission in its own right.

Side-Mission

A quick deep-dive into Microsoft's Docker .NET Docs made me think I only need the following docker image:
mcr.microsoft.com/dotnet/aspnet:6.0

So packing everything in a simple Dockerfile


FROM mcr.microsoft.com/dotnet/aspnet:6.0-focal


WORKDIR /DotNetVoyage

COPY ./appsettings.json /DotNetVoyage/Server-Files/

# Copy every '.dll' file I was handed 
COPY ./* /DotNetVoyage/Server-Files/
CMD [ "dotnet", "Server-Files/Server.dll" ]

Building it locally and accessing the App initially on the dedicated ports seemed to work just fine and I thought
I was getting off work a bit early today. I jinxed myself.

Upon some API testing of the image I started seems some logs that were strange to my non .NETian brain.

The specified framework can be found at:
It was not possible to find any compatible framework version
The framework 'Microsoft.AspNetCore.App', version '5.0.0' (x64) was not found.
 - The following frameworks were found:
    6.0.6 at [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
You can resolve the problem by installing the specified framework and/or SDK.

I was head scratching as to how I can make my runtime which is 6.0.0 potentially have an SDK version that must
have 5.0.0 SDK on it. It got worse, I found out I needed the now defunct and not supported 2.1 version of the SDK
too!

Build everything from scratch?

Unaware if there was a way to introduce the different SDKs into the already slimmed container image, I had lost hope
and decided to build an image with maybe debian or ubuntu or alpine.

However, upon asking the Slack Community and rummaging through some StackExchange queries, I traced out a plan that was
pretty familiar to me: Using Multi-Stage Docker Builds.

Charting towards a Destination

I use Multi-Stage Docker builds almost everyday for work and so I decided to design my container image as follows:

Stage 1: pull SDK 2.1
  1.1: find out which directory do the SDK Files persist

Stage 2: pull SDK 5.1
  2.1: find out which directory do the SDK Files persist

Stage Prod: Pull the 6.0.0 Runtime
  Prod.1: copy all the SDK Files (from Stage 1 and Stage 2) to the same directory here in Prod Stage
  Prod.2: copy all the DLL files from host and run the dedicated DLL

Where art thou SDKs?

A quick pull:

docker pull mcr.microsoft.com/dotnet/sdk:2.1

and check within the container:

docker run -it --rm --name=sdk21-check mcr.microsoft.com/dotnet/sdk:2.1 dotnet --list-sdks

resulted in:

2.1.818 [/usr/share/dotnet/sdk]

Gotcha! the directory for the SDKs is /usr/share/dotnet/

Multi-Image Image

what I mean by this is whether Docker is smart enough to directly pull an image either using COPY or ADD
instructions without me having to create Stage 1 and Stage 2 using syntax such as:

FROM mcr.microsoft.com/dotnet/sdk:2.1 as base21

FROM mcr.microsoft.com/dotnet/sdk:5.0 as base50

It turns out that COPY is extremely flexible! with the COPY --from instruction in the Dockerfile
is completely capable of pull the image from a registry and the dedicated directories can be copied easily!

Solution

Final Checks for Dockerfile

FROM mcr.microsoft.com/dotnet/aspnet:6.0-focal

# Add those missing SDKs here
## Syntax: COPY --from=<registry/image:version> image_directory ProdStage__dest_directory
COPY --from=mcr.microsoft.com/dotnet/sdk:2.1 /usr/share/dotnet /usr/share/dotnet/
COPY --from=mcr.microsoft.com/dotnet/sdk:5.1 /usr/share/dotnet /usr/share/dotnet

# List all the SDKs / Runtimes available in the container
CMD ["dotnet", "--list-sdks", "dotnet", "--list-runtimes"]

build the image and run it and the output will be similar to:

Microsoft.AspNetCore.All 2.1.30 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.30 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.30 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Voilá ! Runtimes v6.0, v5.0, v2.1 all in the same container !!

Final Dockerfile

FROM mcr.microsoft.com/dotnet/aspnet:6.0-focal

# Add those missing SDKs here
## Syntax: COPY --from=<registry/image:version> image_directory ProdStage__dest_directory
COPY --from=mcr.microsoft.com/dotnet/sdk:2.1 /usr/share/dotnet /usr/share/dotnet/
COPY --from=mcr.microsoft.com/dotnet/sdk:5.1 /usr/share/dotnet /usr/share/dotnet

WORKDIR /DotNetVoyage

COPY ./appsettings.json /DotNetVoyage/Server-Files/

# Copy every '.dll' file I was handed 
COPY ./* /DotNetVoyage/Server-Files/
CMD [ "dotnet", "Server-Files/Server.dll" ]

Dedicated StackOverflow Query I created and answered

Just another day on the decks with Docker !!

Get in touch if you feel like dropping on suggestions, feedbacks or criticism!

Use htpasswd to secure your node-RED Container

Shan Desai — Thu, 09 Jun 2022 21:10:01 +0000

Securing Node-RED

node-RED containers can be made secure when using Admin Authorization features
using its settings.js file and mounting it to the container. Sounds quite simple,
however unlike other containers that let your configure their environments via environment
variables whereby the stored credentials are in plaintext, node-RED passwords typically are
stored in encrypted format i.e., bcrypt.

Automation: Encrypted Passwords

There is a possibility of obtaining the hashed (encrypted) password using an ephemeral container
using:

docker run -it --rm --entrypoint /bin/bash nodered/node-red:2.2.2-12-minima

One in the container use:

bash-5.0$ node-red-admin hash-pw

which will prompt you to enter the password and the resultant hash password will be available for you
to use.

This is however, a bit more of manual work and in case where you wish to perform more easy, automated
setups we might need to find a different way.

Automation Script via `htpasswd`

Apache provide a great CLI tool called htpasswd that can encrypt our plaintext passwords
into the required bcrypt hashed format which the node-RED UI and Admin Auth backend will decrypt.

For most of the distributions the package name is apache2-utils so for Ubuntu/Debian you can install
htpasswd using:

sudo apt-get install apache2-utils

Jumping right into it!

You can go through the man pages of htpasswd to find how it might suit for your needs. I am cutting
to the chase as to how I used it.

In order to generate my hashed password that will be compatible with node-RED's decryption backend, I had
to find the total Computation Cost (rounds) needed, which I found out the value was 8 based on some
code diving using node-red-admin repository.

htpasswd -nb -B -C 8 admin testpassword

will produce the following result:

admin:$2y$08$JTOJDi/whZxz8qwsdwCkfedhVOhh00QiKHICdHLImGS8XM2v.fng2

Creating an Environment Variable Template file for node-RED Docker Container

we will make the automation a bit more easier for us, whereby we can use envsubst commandline to fill
and environment variable called NODERED_AUTH_ADMIN in a file called node-red.env. But in order leverage
the power of envsubst we will create a template file called node-red.tpl.env which has the following
content:

cat node-red.tpl.env
NODERED_AUTH_ADMIN='${NODERED_AUTH_ADMIN}'

We create simple bash script which will do the following:

Accept a plaintext password as an argument to the script
use this plaintext password and hash it with htpasswd
export the variable which has the result of htpasswd
use the exported variable to replace the node-red.tpl.env file to node-red.env with the value
unset variable

The script is as follows:

cat nodeREDPwgen.sh
#!/bin/env bash

function nodeREDPW {
    # 1st argument to this script is plaintext password
    local NODERED_AUTH_ADMIN=$(htpasswd -nb -B -C 8 admin "$1")
    export NODERED_AUTH_ADMIN
    touch node-red.env
    envsubst '${NODERED_AUTH_ADMIN}' < node-red.tpl.env > node-red.env
    unset NODERED_AUTH_ADMIN 
}

nodeREDPW $1

Example Usage:

chmod +x nodeREDPwgen.sh

./nodeREDPWgen.sh testpasswd

will produce a node-red.env file after successful execution with the following content:

NODERED_AUTH_ADMIN='admin:$2y$08$/8NVmF2gJ3JDrmv0tX2ud.P7S9gqvB0ds2cRByhnZcRdZOWhTnPN.'

Creating a `settings.js` file for Node-RED Container

copy the settings.js file from the node-red repository and adapt the adminAuth object
as follows:


  adminAuth: {
    type: "credentials",
    users: [{
      username: process.env.NODERED_AUTH_ADMIN.slice(0, process.env.NODERED_AUTH_ADMIN.indexOf(':')),
      password: process.env.NODERED_AUTH_ADMIN.slice(process.env.NODERED_AUTH_ADMIN.indexOf(':')+1)
    }]
  },
}

Creating a `docker-compose.yml` file

services:
  node-red:
    image: nodered/node-red:2.2.2-12-minimal
    container_name: secure_nodered
    env_file:
      - ./node-red.env
    ports:
      - "1880:1880"
    volumes:
      - ./settings.js:/data/settings.js

bring the container up:

docker compose up -d

The UI is available at http://localhost:1880

Problem

If you try to login in by typing testpassword in the Admin UI, you will get Login Failed! But Why?

Investigations

It turns out that the password generated by htpasswd produces a variant of bcrypt $2y where as,
if we were to generate the password using node-red-admin hash-pw and added testpassword it would
generate a completely different looking password with the variant $2b.

Example: $2b$08$sfPlvPNDVVz/duAbAC2bbuNEVuvrwmU01x7plOAQIem6H187Xxq9G

Solution

It turns out that the $2y variant of bcrypt has compatibiltity and we could simply try and replace
$2y to $2b in our settings.js as follows:

module.exports = {
  // .. removed for brewity
  adminAuth: {
    type: "credentials",
    users: [{
      username: process.env.NODERED_AUTH_ADMIN.slice(0, process.env.NODERED_AUTH_ADMIN.indexOf(':')),
      password: process.env.NODERED_AUTH_ADMIN.slice(process.env.NODERED_AUTH_ADMIN.indexOf(':')+1).replace("$2y$", "$2b$"),
      permissions: "*"
    }]
  },
  // removed for brewity
}

notice the .replace("$2y", "$2b") at the end of password key.

Let's spin the container up again using docker compose up -d and try to login with testpassword et Voila!!

We are able to login into Node-RED!

I found that Nick O'Leary (founder of Node-RED and the great guy that he is..) already did some research on this,
I just had to bring this all together.

If you have some suggestions, critiques than please get in touch with me and I would be happy to help.

Resources

GitHub Gist of the files for the impatient ones!

Set the hostname of a Docker Container same as that of your host machine

Shan Desai — Wed, 08 Jun 2022 19:02:30 +0000

Setting Container Hostname to be same as your Host Machine

Sounds trivial, but it turns out you need to do some tweaks in order to set the hostname
to be the same as that of Host Machine.

Scenario

I had a requirement at work, where I needed to set the hostname of a specific container
to be that of the Machine it was supposed to run on.

The Linux Distribution under consideration was Ubuntu 20.04 LTS.

Ubuntu's mysterious `HOST` and `HOSTNAME` environment variables

If you are on an Ubuntu machine right now, try echo $HOS and press the TAB key to let the bash completion fill out. High chances that you will see HOST and HOSTNAME
as available Environment Variables already available to your bash shell's session.

Fairly Simple, you could simply use either one of them in your Compose file as an environment variable to the hostname key and should work out of the box! Not quite!

Investigation via a simple Example

Take the following docker-compose.yml file

services:
  test:
    image: alpine:latest
    container_name: hostname-tester
    hostname: alpine-${HOST}
    command:
      - hostname

The following test service should be able to retrieve the HOST variable value and set it as the hostname of the alpine container. The command should be able to print something like alpine-my-ubuntu as an example.

Let's see what happens when we run docker compose up

WARN[0000] The "HOST" variable is not set. Defaulting to a blank string. 
[+] Running 1/1
 ⠿ Container hostname-tester  Recreated                                    0.1s
Attaching to hostname-tester
hostname-tester  | alpine-
hostname-tester exited with code 0

As you see the HOST variable although available to the shell is not available within the
Compose file.

Most of the shell environment variables are visible via using printenv on the host machine, so a quick search for HOST or HOSTNAME reveals that these specific environment variables are not in the env of the shell

printenv | grep -i "host"

Solution

The most standard way to make a variable available for a shell session is by exporting it.

Let's export HOST using:

export HOST

Let's try bringing the container back up again and see if it picks up the variable values

❯ export HOST
❯ docker compose up
[+] Running 1/0
 ⠿ Container hostname-tester  Recreated                                    0.0s
Attaching to hostname-tester
hostname-tester  | alpine-shan-pc
hostname-tester exited with code 0

for my machine (Manjaro Linux (Rolling)) I could now set the hostname of the container to be the same as that of the Host Machine.

Sure enough searching through printenv again you will find the export HOST variable.

If you have on-prem servers or cloud images where you might need such a configuration a solution would be to add export HOST to your user's ~/.bashrc or ~/.zshrc (if using ZSH)
and the value will be available when bringing the corresponding Compose Stack up.

Hope this information helps people trying to find a similar solutions when it comes to Docker and Docker Compose based environments.

No Setup Development: Productivity Experience with Docker

Shan Desai — Sun, 08 May 2022 09:13:47 +0000

Why I stopped worrying about setting up environments!

If Stanley Kubrick were a Software Engineer, he would have named this post

Dr. Nosetup: How I Stopped Worrying About Setting Up and Love the Development

(I'll see myself out with that pun!)

I tried contributing to an open-source project without actually setting up the
complete programming language tools, and it felt like worth documenting.

Problem: So much to download, and setup before getting to work

I tried sending a feature to the node-red GitHub Repository with a new TOML configuration node.

However, I didn't want to taint (pardon me for using the word) my personal laptop by installing
node.js and npm.

One particular reason being, is that I have less time now to continue with Web Development stuff,
and node.js isn't my preferred language anyways. I want my host laptop to be as minimal as possible.

But I wanted to send the feature patch upstream because I was in the zone.

Solution: Docker encapsulated Environment

Since I have been heavily using docker for a while now, I asked myself

What do I need to send a patch upstream?

A: Only relevant files

Does docker provide me a node.js environment?

A: Yes, certainly it does. Not just node.js but for all possible programming languages

How do I avoid doing manual copy-paste labour for files between the container and my laptop?

A: Volume-Mounts. Any changes within the containe get reflected back to the host laptop and vice-versa

Setting it Up!

All I really needed was docker on my machine and we are ready to go!

Steps:

clone the repository to dedicated directory on my host laptop
Visit Docker Hub and find the node-js Image repository
Find the Long-Term-Support (LTS) version image tag. In my case it was 16.15.0

So we almost have everything we want!

Caveats

Remember that Docker Containers are their own ephemeral worlds all together.

If the containers are designed to with root users, your files may change ownership, or might have
different owners. You could check this using ls -la in your directory.

I really want to avoid such scenarios, such ownership issues might affect your filesystem as well as
the upstream code. But no issues, docker CLI provides way to control the user and group settings
before bringing the container up.

It is also worth mentioning that, the container environments also produce files that should be not be
reflected in your commits upstream. In the case of node-red the package-lock.json is a file created
within the container that will be mapped to the host machine.

It might be wise to keep such files into .gitignore as well as .dockerignore files within the development
repository to avoid accidently committing them upstream or bringing them within the container.

Docker CLI

$ # assuming your are in the development repository
$ docker run -it --name=node-red-TOML \
     -u $(id -u):$(id -g) \
     -v $(pwd):/usr/src/app \
     -p 1880:1880 \
     node:16.15.0 \
     /bin/bash

The -u parameter maps your current user id and group to the container, avoiding any root ownership
conflicts.

The -v parameter is the volume mount that will map the codebase to the /usr/src/app directory in the
container.

There you have it ! A node-js environment without having to download and setup the tool on the host!

You can now code everything with ease with your Editor of your choice with the container running.

Any changes, either on the host or within the container will be reflected to your editor.

Just make sure to run the execution commands in the container.

Benefits

This worked out well for me! I was able to get the codebase up and running in no time, without having to
worry about incompatibility issues.

Changes made in my editor (new files, refactored files) are available in the container to use and execute.

Running the commands within the container makes it easier to know what happens and all of this is ephemeral
so I don't have to do a lot of cleanup afterwards.

Just remove the container and commit the code!

On a side-note, the feature patch upstream wasn't required by the core team :(, but I could use the
same development environment pattern to create a node-red-contrib node. So nothing does to waste!

Hope this helps, get in touch if you would like to provide some suggestions, criticisms!

Create a custom Node-Red Docker Image and performance Integration Tests and CI/CD Pipelines

Shan Desai — Sun, 10 Apr 2022 11:48:15 +0000

Pre-Requisites

This post assumes that the reader has an understanding about Node-Red Flow
Programming tool and wishes to work with a custom Docker Images for it.

This post also assumes the readers are familiar with python programming.

Requirements

Since Node-Red provides a visual way of connecting many components through Flows,
we might be interested in using it for our own personal stack.

If you haven't tried it out, Node-Red Documentation for Docker provides some nice
ways to getting started.

Since the Flows are rendered to a web-browser, we might want to take into consideration
testing our custom Node-Red container when it might be integrated with other containers
within a stack deployment.

This guide will provide a concise way of joining the dots between creating your custom
Node-Red container and testing its reachability within a docker-compose file using
the pytest-docker-compose suite.

Follow Along

Please refer to the GitHub Repository for
the code structure and file content

Custom Node-Red Docker Container Creation

Diving into the deep-end straight, we want to create a Node-Red custom container with
just the following:

node-red runtime-engine
node-red dashboard

NOTE: for this post sake, we stick to basic things, please feel free to add more things
according to you needs.

Adding Node-Red and its Dependencies

Standard practice with Node-Red lets the user add dedicated flows and dependencies via the
npm install command-line. However within the Node-Red-Docker Repo Wiki, you can also
add these flows/dependencies via a dedicated package.json.

Let's stick to this method and avoid adding things via CLI within the container.

Create a package.json file and add the following:

{
    "name": "node-red-slim-container",
    "description": "A Slim Node-RED Docker Image running on Alpine Container",
    "dependencies": {
        "node-red": ">=2.2.0",
        "node-red-dashboard": "*"
    },
    "scripts": {
        "start": "node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS --userDir=/data"
    }
}

Moving on to creating a Dockerfile

Multi-Stage Dockerfile for a Slim Image

We will be using a Multi-Stage Docker build process to keep our image a bit light on the
size using alpine based containers and the official minimal image from Node-Red.

We will use TWO-Stages in our build process:

prepare a base Image with alpine
use nodered/node-red-minimal image to install the flows/deps and prepare the node_modules

We then use the base Image in step 1 as a production image and copy the node_modules directory
to this production image.

Our Dockerfile looks like the following:

FROM alpine:3.13 AS base


RUN apk add --no-cache \
            nodejs \
            npm && \
    mkdir -p /usr/src/node-red /data && \
    adduser -h /usr/src/node-red -D -H node-red -u 1000 && \
    chown -R node-red:node-red /data 

FROM nodered/node-red:2.2.2-minimal AS build

COPY package.json .

RUN npm install \
        --unsafe-perm --no-update-notifier \ 
        --no-audit --only=production

FROM base as prod

COPY --from=build --chown=node-red:node-red /data/ /data/

WORKDIR /usr/src/node-red
COPY settings.js /data/settings.js
COPY flows.json  /data/flows.json

COPY --from=build --chown=node-red:node-red /usr/src/node-red/  /usr/src/node-red/
USER node-red

CMD ["npm", "start"]

We start with the base image as alpine:3.13 and prepare it by adding nodejs and npm
needing to run Node-Red. For security best-practices, we add a node-red user and group as
opposed to running everything as root. This avoids any privilege escalation within the container,
as well as misuse within the container.

Our build stage will install all the NPM packages in our package.json and then we copy the
necessary node_modules to our prod image. In the end, we will run the command as npm start

You can build this image locally using:

docker build -t node-red-slim:latest .

Add a Healthcheck API in Node-Red

You can refer to this Discussion Thread on Node-Red Forum to create the flow, or use the following
flows.json file

We can check if the flow is working again but building the Docker Image and then try:

    curl -XGET http://localhost:1880/health

Don't forget to set the port 1880 when running the docker image e.g.

    docker run -p 1880:1880 node-red-slim:latest

If that works, good job! now it is time to make an integration test with python

Integration Test of your Node-Red Image

One cool thing you can try out is the pytest-docker-compose which provides you a nice suite
to do tests with docker-compose. I won't dive too deep into it, but I relied on this
Dev.To post by Iuliia Volkova for wrting a simple test that does a health check on our
/health API once the container is up and running.

Testing Files

pytest-docker-compose is a plugin for pytest testing suite. The structure of your test directory
should be as follows:

.
├── README.md
├── requirements.dev.txt
├── scripts
│   ├── 00-build-test-env.sh
│   └── 01-run-tests.sh
├── test-docker-compose.yml
└── tests
    ├── conftest.py
    └── test_fixtures.py

Refer to tests/integration directory for file contents

The conftest.py is used to as fixture for the test-docker-compose.yml to be used for the test stack.

the test_fixture.py contains the test fixture to ping the /health API from our flow once the
node-red container

For our case, if the /health API provides a HTTP 200 OK response, our test passes. You can add more
tests depending on your requirements.

We will leverage simple bash scripts in the scripts directory to setup the python
virtualenvironment as to bring the test-docker-compose up and run the pytest

Additional: using this setup in a CI/CD Pipeline using GitHub

You can think of our steps we just went through as a CI/CD Pipeline. The steps are:

Build our custom Node-Red Docker Image
Test Integration using pytest-docker-compose
Push the image to Docker Hub, if test passes

For the repository, I have setup a GitHub Workflow that pushes the docker image only
when I push a Tag e.g. v1.2.0 etc.

For this Workflow, you will need to setup the credentials of your Registry in your GitHub Secrets.
For a sample of the YAML file, refer to this deploy.yml

Inference

That's how you do custom node-red, docker, docker-compose and integration testing !

I am not aware of conducting tests of the individual tests of the flows generated in Node-Red, but this
test focuses on integrating node-red with some possible stacks where different Databases are part of the
stack.

If you have comments, criticisms, feedback please reach out to me and I will be happy to learn and help!

Prepare your Docker-Compose Stack as a Tarball for Offline Installations

Shan Desai — Sat, 26 Mar 2022 13:52:49 +0000

Requirements

Many a times you might have requirement to run Docker on a Raspberry Pi or other SBCs that may have to do run some applications without any internet connections. You might have initially developed your application only by keeping in mind a Docker Artifact and was meant to run as complete stack using docker-compose.yml.

In almost all cases, bringing the stack up will pull the images from a registry through the Internet. But when you wish to deploy some SBCs into a space where there is no Internet to begin with you might have to either develop your code artifact as a package or binary bundle. If this application requires dependencies on some Databases then you will have to write tedious bash scripts to prepare the device
for some pre-installtion.

Seems like a lot of refactoring / re-thinking to do!

Do not worry docker has things sorted out, and provides docker save and docker load functionalities.

Scenario

Let's consider an Open-Source Stack tiguitto for IoT at Edge.

It comprises of some standard apps like an MQTT Broker, InfluxDB, Telegraf, Grafana. All these
components are combined through a docker-compose.yml file.

Now, as previously mentioned shipping the stack would have been easy, had we had Internet on the target node.
Simply sending the compose file to the node would have done everything for us.

For cases where we do not have Internet access, we have to ship two things:

docker-compose.yml
Compressed Tar-Ball for services present in the compose file

Usage Steps

Assume we have our development machine with docker and docker-compose on them.

We make sure to remove all not-used, unnecessary images are cleared from the machine and only the images of our application exists.

    docker images -a # check existing images on dev machine

If you wish to remove everything and only pull the required images do:

    docker rmi $(docker images | awk "NR>0 {print$3}")

Pull the images with their dedicated tags using docker pull <image_name>:<tag>
Check using docker images -a

Now leverage docker save along with some format parsing using:

docker save -o myStack.tar $(docker images --format "{{.Repository}}:{{.Tag}}")

docker save combines the image into their respective Tar balls however we need to tell docker that we need to combine all the images existing on the dev machine into a combined Tar ball. This is done through --format "{{.Repository}}:{{.Tag}}"

This should produce myStack.tar in your directory

Unpack your Tarball and Test

If you are doing this on a laptop or PC, try switching your network connection off now.

Unpack your Tarball

    docker load -i myStack.tar

Check with docker images -a
You can now docker-compose up and your application should work offline

If you use some other ways of using shipping your Docker and Compose applications, get in touch with me and would like to learn from you!

Use Docker Secret to set your Portainer Admin User's Password

Shan Desai — Sun, 13 Mar 2022 10:57:31 +0000

Requirements

I was deep-diving on creating a decent-sized docker-compose stack with 5+ services in them.
I was also on the look out for some Deployment Pattern where I didn't want to dump all my
Environment Variables into a giant .env file, neither in Production nor in Development.
I decided to create a structure where each service aka. component would have their own dedicated
directory with their respective secrets (credentials) and their configuration files all bundled
together.

I decided upon the following structure:

ComponentName
├── config.component
└── secrets
    └── .env.component

Here the ComponentName could your favourite database, the secrets directory within this directory
would contain a .env.database file with all necessary admin credentials for the initial stack configuration.
The config.component file is a dedicated configuration file (if needed)

I think the structure would work Okay since changes to a component's configuration would be reflected
a bit better in the Git History / Logs

Problem

If you end up having a lot of containers in the stack, docker-compose logs only takes ever so far! You might
decide upon Portainer to make the container management a bit easier through it's UI.

One problem, Portainer doesn't support the Environment Variables Configuration pattern as by many container
based software applications.

It relies on executing command in your docker-compose file to introduce an password into your Portainer's service.

If you have a bit of Software OCD, you are going to hate it! One service not conforming to your dream stack deployment!

Approach

Here comes docker secret to the rescue!

One massive Caveat, according to the documentation, docker secret is only available for Swarm Setting, or it ?!

A blog post from Michael Irwin about using
docker secrets in Development provides a way to trick docker into accepting files as a docker secret and use it into your
service.

Another thing when going through Portainer's Documentation is that, it accepts admin password also through a
password file

We could see some Puzzle Pieces falling into place here!

Solution

Let's start with our structure mentioned above:

portainer
└── secrets
    └── portainer-admin-password
|- docker-compose.yml

Instead of an .env we rename the file to portainer-admin-password where you can set your password.

Your docker-compose.yml will then look as follows:

version: '3.7'
services:
  portainer:
      image: portainer/portainer-ce:latest
      container_name: sys_portainer
      restart: always
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
      command:
        - '--admin-password-file=/run/secrets/portainer_admin_password'
        - '--no-analytics'
      networks:
        - some_network
      ports:
        - "9000:9000"
      secrets:
        - portainer_admin_password

networks:
  some_network:
    external: true

secrets:
  portainer_admin_password:
    file: ./portainer/secrets/portainer_admin_password

Breaking it down functionally!

secrets:
  portainer_admin_password:
    file: ./portainer/secrets/portainer_admin_password

mounts the file as a docker secret into the compose file. Since it is a file mount, it will be
available in the following directory /run/secrets/<KEY_NAME_IN_SECRETS_section> since our key
is the same name as that of the file (in order to avoid any discrepancy), the content of the file
will be consumed by the --admin-password-file flag during the stack initialization.

Voilá! You now have a similar structure without having to hardcode passwords or sensitive information
into your docker-compose.yml file

Seed Your MongoDB Container with JSON data using Docker and Docker-Compose

Shan Desai — Tue, 09 Nov 2021 12:21:07 +0000

Requirements

A couple of years ago I made a frontend app which rendered JSON data stored in-memory. The App provided a check-list for Quality Management called QualiExplore (under Apache 2.0 License).

Nothing too fancy, but we needed to make the app a bit more flexible and decided to use MongoDB to render the information
as well as make it easy to introduce new information to the JSON data that previously persisted in the App.

Fundamentally, the app should be out-of-the-box usable for someone who wants to use it (with other additions currently on-going e.g., GraphQL backend + User authentication / authorization)

Problem

The JSON data, that previously persisted in the app needs to be seeded into the MongoDB instance once the complete stack is
brought up.

Doesn't sound like a bit deal, should be possible to use volume mounts with docker, right? Yup, you guessed it but MongoDB doesn't
just ingest data because one mounted a volume to a specific directory. To the rescue comes mongoimport tool where one can load
data via CLI. Seems like our problem is solved!

Well it turns out I wanted to seed data into the MongoDB in a litte more secure manner aka. maybe use a username and password and a dedicated initialized database where this imported data should persist under specific collections.

Turns out, once you figure out the mongoimport tool's Syntax, the next big headache was to figure out how to securely pass this information to a docker image which will insert this information into Mongo.

I generally have a stack design pattern where I try to keep changes into YAML files for an end-user minimum and let the changes reflect through the environment variables passed to the stack. So passing the environment variables to the respective docker containers shouldn't be a big hassle, right? Oh! how wrong I was!

Approach

I decided to following the following sequential steps:

Bring the MongoDB container up
Create a docker container that is connected to the same network as MongoDB container as above that has mongoimport on it
Mount the JSON data into the container in step 2 and insert the data
In order to make the ingestion of data bit more secure pass the information like the database's URI, credentials via environment variables

So the initial project setup is as follows:

.
├── docker-compose.yml
├── mongo_seed
│   ├── Dockerfile
│   ├── factors.json
│   └── filters.json
└── .env

Here, the mongo_seed is the directory with the data to be ingested as well as the Dockerfile that will perform the necessary ingestion.

The root of the directory contains the docker-compose.yml for the stack and its respective .env file for environment variables

Problems Faced

This requires a quick primer of Docker's ARG and ENV in context of docker. A decent read-through I used can be found
here

In a nutshell

ARG is used during build-time i.e., when building a docker container. It's lifetime is short till the build is complete. Once the container
is built, they are not available for further usage.
ENV, on the other hand, is available for the run-time of the container too.

A common practice that a lot of container developers use is to declare an ARG and then use the value of the incoming build-argument as an
ENV for further usage.

so something like

ARG database-uri

ENV DATABASE_URI=$(database-uri)

# Use `$DATABASE_URI` to insert database

This wasn't the case since I ended facing problem which I documented on
StackExchange

The problem concisely, is the environment variables from .env fille passed into the mongo-seed container doesn't get set during run-time

Solution

After going through some StackExchange Queries, an answer from
Zincfan on a Query regarding getting Env Variable in Dockerfile turned out to be the solution

Steps

Declare Same ARG and ENV of the same name as that of the environment variables
Get the information from the build-argument and store the information into the env. variable using the syntax:
```
```
ARG DATABASE_URI
ENV DATABASE_URI ${DATABASE_URI}
```
```
use ${DATABASE_URI} in the run-time of the container

to pass the information from the .env passed to the compose file should be in the following way:

 services:
     mongo-seed:
         args:
              - DATABASE_URI=$DATABASE_URI

Files

So finally my Dockerfile looks like the following:

FROM mongo:5.0
 # Will be set through Environment Files
 ARG DATABASE_URI
 ARG USERNAME
 ARG PASSWORD

 ENV DATABASE_URI ${DATABASE_URI}
 ENV USERNAME ${USERNAME}
 ENV PASSWORD ${PASSWORD}

 COPY factors.json /factors.json

 COPY filters.json /filters.json

 CMD mongoimport --username ${USERNAME} --password ${PASSWORD} --uri ${DATABASE_URI} --collection factors --drop --file /factors.json && \
     mongoimport --username ${USERNAME} --password ${PASSWORD} --uri ${DATABASE_URI} --collection filters --drop --file /filters.json

Good to Note: Never split into two CMD in the file, rather merge multiple commands into one CMD.

my docker-compose.yml file looks like the following:

services:
    # MongoDB
    mongo:
        container_name: mongodb
        image: mongo:latest
        env_file:
            - .env
        ports:
            - "27017:27017"
        networks:
            - "qualiexplore_net"

    # Initial Seed to QualiExplore Database
    mongo-seed:
        env_file:
            - .env
        build:
            context: ./mongo_seed
            dockerfile: Dockerfile
            args:
                - DATABASE_URI=$DATABASE_URI
                - USERNAME=$MONGO_INITDB_ROOT_USERNAME
                - PASSWORD=$MONGO_INITDB_ROOT_PASSWORD
        depends_on:
            - mongo
        networks:
            - "qualiexplore_net"

My .env file is as follows:

MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD=example
MONGO_INITDB_DATABASE=qualiexplore
DATABASE_URI=mongodb://mongodb:27017/qualiexplore?authSource=admin****

Comments, Suggestions

If you would like to suggest me some changes in my design or a better way to tackle this situation, I am all ears
and would love to improve and discuss aspects. You can contact me on LinkedIn or or leave a comment here

Setting up and running LF-Edge's Fledge (from source) on a Raspberry Pi 4

Shan Desai — Fri, 30 Jul 2021 10:03:35 +0000

Fledge

Fledge is an open source framework and community for the industrial edge focused on critical operations, predictive maintenance, situational awareness and safety. Fledge is architected to integrate Industrial Internet of Things (IIoT), sensors and modern machines with the cloud and existing “brown field” systems like historians, DCS (Distributed Control Systems), PLC (Program Logic Controllers) and SCADA (Supervisory Control and Data Acquisition). All sharing a common set of administration and application APIs.

I wanted to give Fledge a try to see how one can bring an open-source IoT/IIoT Stack to a quick realization.

Getting Hands Dirty already!

Hardware

Device: Raspberry Pi 4 Model B
Spec: 2GB RAM, 32GB Micro SD Card
Operating System: Ubuntu LTS 20.04.2

Setup

I am going to just clone stuff in my home directory

Clone the fledge Repository

$ git clone https://github.com/fledge-iot/fledge.git && cd fledge/

Install the Dependencies (in the root directory of fledge)
```
 $ sudo ./requirements.sh
```
Let's install fledge on the Pi
```
$ sudo make install
```
grab something to munch on till the installation is complete
The installation will add the files and binaries in to the following directory: /usr/local/fledge/. You can export this path:
```
$ export FLEDGE=/usr/local/fledge
```

start Fledge and check the status using the following:

$ $FLEDGE/bin/fledge start
$ $FLEDGE/bin/fledge status

Checkout the Fledge UI by hitting the Pi's IP Address: http://<IP_ADDRESS_of_PI>

At this point of time, Fledge is very bare-metal which implies there are lot of plugins that are missing. We begin by installing a very basic simulated plugin.

South-Plugin Installation

Sinusoid

Since we are doing things by source, we need to clone the plugin repository. We start with a very basic sinusoidal South Plugin which will generate sine waves to the Fledge Core.

clone the fledge-south-sinusoid plugin repository

$ cd 
$ git clone https://github.com/fledge-iot/fledge-south-sinusoid.git && cd fledge-south-sinusoid/

we just need to copy the source code of the sine-wave generator to the directory path of plugins for Fledge

$ # we are currently in the root directory of fledge-south-sinusoid
$ sudo cp -r python/fledge/plugins/south/sinusoid $FLEDGE/python/fledge/plugins/south

You can refresh the UI and click on the South Menu and you will be able to see sinusoid as a South Plugin

CoAP

Let's get one more Plugin into Fledge: CoAP plugin

clone the fledge-south-coap plugin repository

$ cd 
$ git clone https://github.com/fledge-iot/fledge-south-coap.git && cd fledge-south-coap/

Let's copy the plugin's code to Fledge as we did previously

$ # we are currently in the root directory of fledge-south-coap
$ sudo cp -r python/fledge/plugins/south/coap $FLEDGE/python/fledge/plugins/south

Copy the python dependencies that are necessary to execute the plugin and install them:

$ sudo cp -r python/requirements-coap.txt $FLEDGE/python/
$ sudo pip3 - r $FLEDGE/python/requirements-coap.txt

Since the code is in the /usr/local/ directory we need to provide root privileges to the coap plugin for execution by the Fledge Core
```
$ sudo chown -R root:root $FLEDGE/python/fledge/plugins/south/coap
```

That's it ! Now you have we two South Plugins in our Fledge Application that can provide us some simulated data!

Your UI South Menu should look like the following:

Setup / Data Generation for Sinusoid Plugin

Let's setup the Sine-Wave Generator by naming it test-sine-south and click Next
Name the Asset Name. We name it sine-wave-asset
Make sure to enable the plugin by checking the Enabled checkbox and finally click Done

On the UI you now see the Sinusoid Plugin Enabled with active readings being fed into the Fledge Core. You can click on the link test-sine-south to change the configuration of the plugin / delete it or export readings to a file.

You can click on Assets & Readings Menu and click on the Graph Icon to get some visualization delete it or export readings to a file.

And there you have visualization of the data coming in from your South Plugin, a lovely sine wave. Not practical but you get to understand how the Fledge Architecture would work!

You can click on the Summary tab to get some basic statistics of the information

Setup / Data Generation for CoAP Plugin

Let's setup the CoAP South Plugin similarly to Sinusoid by clicking on the + Add button on the top-right of the Panel under South Page
Call the service test-coap-south and click Next
In the next step keep the configuration as it is for now i.e. Port as 5683 and the URI as sensor-values and click Next and enable the plugin on the last setup.

This should enable the CoAP Plugin for Fledge and it's time to generated some CoAP data from a simulated client to a CoAP Server running on the Pi.

Fledge does provide some testing information for CoAP in the core directory under the the CLI called fogbench

Let's generate some CoAP data on the Pi.

$FLEDGE/bin/fogbench -t $FLEDGE/data/extras/fogbench/fogbench_sensor_coap.template.json -I 100

here -t flag is for mentioning a template and we use a coap template provided by Fledge mentioned above. -I is the time interval

Upon executing the command on the Pi, you can check the UI which will generate the following output for the CoAP Plugin.

You can explore the Assets and Visualization by going to Assets & Readings and clicking on any of the asset you wish to observe.

NOTE: In order to visualize live-data you can run the fogbench command above a couple of more times.

Filter Plugins in Fledge

The sine wave isn't that cool to look at to so let's get some Frequency Domain Analysis on it with a Fast Fourier Transform on it using Fledge's FFT filter plugin

Let's clone the plugin in the home directory

$ cd 
$ git clone https://github.com/fledge-iot/fledge-filter-fft.git && cd fledge-filter-fft/

This plugin is not python based so we need to do some tasks for installing it. We export a FLEDGE_ROOT variable and create a build directory in the root of the repo

$ # we export the directory where we built Fledge for the filter plugin to use
$ export FLEDGE_ROOT=~/fledge/
$ # in the root of the `fledge-filter-fft` repo
$ mkdir build && cd build/

compile the plugin
```
$ cmake ..
```
install the plugin
```
$ sudo make install
```

Now you have the FFT plugin ready to be applied in Fledge

Applying FFT on the our Sine-Wave Asset

Let's head back to the South Plugins in the UI and click on the test-sine-south service
Click on the Applications menu for the Service
Select fft from the available plugins and name the service test-fft-sine
Setup the Service to conduct filtering on the asset called sine-wave-asset (See steps to setup Sinusoid Plugin) and other parameters as shown below (no fine tuning done here). Don't forget to click the Enabled checkbox and finally Done

The Fast Fourier Transform might take some time to show up but you can head to Assets & Readings and click on the sine-wave-asset FFT's Graph Icon to see some visualizations. This is what mine look like

Great stuff. Imagine having vibration sensors and you need to conduct FFT analyses on it! Fledge makes it nifty!!

How to (atleast) not get downvoted on StackExchange or anywhere: My Bug/Query Template

Shan Desai — Thu, 15 Jul 2021 16:36:57 +0000

(Originally posted on my personal Blog)

Disclaimer

Everything mentioned in the blog are my own findings / discoveries to survive the often times brutal down-voting of
StackExchange / StackOverflow Queries. Please feel free to ignore any or all of this if you have a better way of documenting errors / bug reports.

Problem

For someone who does not have an education in Software Engineering but yet loves writing Software, I frequently, if not always, rely on the community of Software Developers / Engineers to help me figure out my problems with programming languages or software suites.

One of the major problems when I started working with code-bases / stacks (in 2015, not so old as I want to make it sound!) was the lack of coherence in trying to frame my problem which I faced whilst working on something. The frustration that mounts on when you are deep-diving into something and hitting a temporary barrier is sometimes baffling.

Now after 6 years of working with software and 2.4k Reputations on StackExchange later, I looked back at why some of my posts keep getting me up-votes even though they were posted a while back!

Bragging Time

At the time of writing this blog, I checked my reputations and I am ranked at:

Top 11% for this year on StackExchange
Top 7% Overall on Internet of Things Beta Community
Top 9% Overall on Rasbperry Pi StackExchange Community
Top 26% Overall on Unix Linux StackExchange Community

One might be tempted to think, that these numbers are weak and do not amount to anything which is absolutely fair to assume. However, please do realize that for someone not with hardcore software background, surviving the down-votes on such communities is an achievement in itself.

Analyses

These are some of the things that have helped me over the years in obtaining answers to queries on forums and also a positive reputation

Mark (this) Down

This is a rather software philosophy that one can debate till the end of time or till the end of a project!

Documentation is not for everyone ! I hardly have met someone who is as much in love with Markdown as I am at this point of my professional career. Don't believe me ? I am writing this blog post in Markdown!

I am mentioning Markdown because it is exceptionally handy to work with. From a blog post such as this, to StackExchange, to different community websites and even Reddit support it in some form or another. Not to mention how minimal it is compared to terrible HTML tags !

It also has some of the lowest learning curves for any Markup Language to my knowledge.

May it be opening an Issue on GitHub, or writing a blog post, or an opinionated post on your favourite subreddit - Markdown is always there to help you transfer your thoughts for you without having to click on that fancy editor buttons !

Start a project with a NOTES.md

you can ignore this file in your software development and use this NOTES.md as a scribbling markdown file for all things you did/ you want / you will need to do in order to get something up and running.

Great thing about making this a markdown file is the flexibility of copying / pasting commands and code snippets that are suspicious.

A CTRL + C and CTRL + V is way comfortable that a million CTRL + Z a lot of times !

Being Concise

This is the hardest thing I had to adapt. In a frustrating bout with some code or a stack, I would have just loved to dump everything that has been through at my screen on a forum and wait for someone to catch my attention. However, life isn't like that and so is software troubleshooting!

Things that can help be concise:

Keep a small notebook to help keep track of basic Command-Lines executed (better yet use history command to know what was done)
If you don't care about jotting things down, try keeping a mindful state when developing in order to trace back some vital steps which may lead to failure.

Depending on your jotting down skills / recollection of your mind's memory you can use this information to summarize what lead you to the problem / bug in the first place and place those information in your NOTES.md for posterity

Document your working system

Not everyone has the same computer / server etc. as yours ! So it is definitely worth keeping track of which OS you have (which build, distribution, etc.) to make it clear to someone that things may be a bit different.

Same logic goes for your programming languages. Each programming language's compiler, interpretor keeps updating itself and it is wise to provide someone willing to help you out with a bit more precise information on how you got started with your work.

Same logic should go for your Dependencies. A lot of programming environments provide a file that keeps track of all your dependencies, and although some are uglier than another (looking at your Java's pom.xml or other .xml files), it is worth mentioning the dependencies that you might have added recently that you suspect might have broken your code

Maybe a git diff can help you get an idea of what was recently added or changed.

Copy / Paste all your URLs

Seems harsh but, if you browsed through some handy tutorial / blog / StackExchange Response, copy the URL in your NOTES.md or bookmark it!

We all suffer from the StackOverflow Copy/Paste Syndrome and in a frustrating event don't realize where and why we changed a bit of our work just get things up and running!

Adding URLs also is extremely beneficially for posterity when someone who wishes to pursue something similar to your work can benefit from it!

Do not write massive paragraphs

My observations over the years is that a lot of communities don't care about a Shakesperean Description of your problems you face ! Instead they prefer to read shorter sentences since they too want to get to the bottom of the mystery that you have posted on.

Granted, this strategy does not always work however it gets the ball rolling and further information can be requested in the comments sections.

Do not Paste your massive code snippet

Let's be honest, not even your QA manager wants to dive deep into your 1000+ lines of code. There is most certainly a certain block of code you feel is the culprit and this can be pasted in the query in order for the readers to only focus on the things that are causing a problem.

An Updated Query Provides More Context

Apropos, if someone asks you to provide more information for your query then a better way to address the additional information is writing it under a new section. A lot of forums provide an Edited View on the query but a lot of times I find them to be let intuitive about the added information.

What got me the Up-Votes?

At this point, you might be like:

Yeah, Yeah ! Show me something substantial and enough with the words already!

Well here is a template I think might help people (new-comers/ beginners) out!

Click on the code snippet below which is already in markdown and add the information accordingly.

Please feel free to add / remove any other information that you might find useful (Let me know so I can improve myself too!)

Template

This is how I have gotten around to structure my queries mostly!

## Host System
- Mention Operating System
    - Mention Build Version / Kernel Version / Distribution

## Programming Language / Software Suite / Software Tool Environment Information

- Compiler / Interpretor Version (mostly available via `--version` flag or `version` parameter)

## Dependencies Graph

- Provide a list of changed dependencies (if any that you think might be the culprit)

## Problem Description / Scenario

Provide the following:

- I want to --INSERT Intent HERE--
- Upon following the step (mentioned below) I end up with
    --- PASTE ERROR LOGS here ---

### Provide a repo overview (if necessary)

Sometimes the repository structure may be helpful to get a better picture 
(Calling modules from different directories etc.)

- Use `tree` command-line to provide a necessary structure of your repository

### Code Snippets (If any, else use Reproduction Steps)

Remember not to dump all the 700 Line code here. 

Add a particular part of the code that you think may be the culprit

Add a note in the end or as a comment in the code snippet `// removed code for brevity` 
to let the reader know that you are focussing on what you think is causing a problem

### Reproduction Steps

Here you can actually make a list of all the command lines
you have executed in order to provide a better context.

A lot of times the error might lie in some of the steps executed here (missing flags etc.)

## Queries / Requirements

Add a list of questions you think you are unable to figure out from the problem 
or a list of requirementst that your code should be able to achieve.

Do not add a massive amount of information here, try to be concise as much as you can.

## Edit (only necessary if requested)

Add a list of updated information here, only if the community requests its.

It is easier to add information here than
rummaging through the comments under the query.

For Posterity and beyond

I am aware that not a lot of my code will be super helpful to people however in terms of Technical Debt, I understand that finding a solution to a problem is not an easy task, however an even harder task is to take the effort for documenting it and making it available for others!
How great does it feel when, you stumble across a query that is very similar to what you are facing and there is an Accepted answer for it!

What also works incredibly well is to easily address the query and the solution in your Software code. You do not have write the complete bug description and the decision to solve the bug would already be mentioned in the URL of the Query. The URLs also work great for a company's Knowledge Base / CMS.

What also works well for me is going back to a particular bug I had previously documented on StackOverflow / StackExchange is quite easy. A few clicks from my user account and voila! I land up to the solution as well as related queries to the topic in no time.

Last but not the least, to be honest, everyone would love to have a quick Dopamine fix when they see an upvoted query or an answer !

Show and Tell

After all this, do I have something to back my claims?

Yes I do!

Here is a list of all the Queries and Answers that have gotten me a lot of upvotes over the years and stick more or less to the template mentioned above:

StackOverflow

Internet of Things (Beta)

Make Embedded Board NTP Server for Arduino Boards in the subnet

Raspberry Pi

Which Model Raspberry Pi am I running

Unix / Linux

If you have tips that you feel I might have overseen or something that needs to be corrected than let me know! I am all about improving!

Provision your Raspberry Pi 4 Cluster with Ansible

Shan Desai — Sat, 08 May 2021 11:50:23 +0000

Originally written on my GitHub Pages Blog

Problem

I have eight Raspberry Pi 4 8GB and wanted to setup them up to be a High Availablity (HA) Kubernetes Cluster for some projects that comprise of Edge, Fog Computing.

The Problem is redundancy of setting these Pis up with pretty much the same thing.

Think of installing something but doing it 8 times in total! Sounds more of an hassle!

Solution

Using ansible we can setup a number of Pis in the cluster without having to repeat anything!

Network

This is what my network look like:

Component Description

Component	Quantity	Description
WLAN Router	1	DHCP Server / Entry for user
Gigabit Switch	1	High-Speed Networking
Raspberry Pi-4 8GB RAM	8	Control / Worker Nodes
64 GB SD Cards	8+	For Operating System / Storage
Cat6e Ethernet cables	8+	For connecting to Switch
Operating System	-	Ubuntu 20.04 - LTS on Pis

Note: SD Cards are not the best choice, since they wear-out over longer usage.

A possible solution is to by USB3.1 based storage disks which can be used to store information
and boot the Raspberry Pi Up.

For the time-being, the SD Cards will do just fine.

Fixed IP Address Allocation

Either use DHCP Address Allocation on your WLAN Router to dedicate a fixed IP address to the MAC Address of your Raspberry Pi 4s or configure your Operating System for your Pis with static IP Addresses. I chose the former method, since mapping the MAC Addresses with defined IP addresses using your WLAN router's UI is way easier. Also I wasn't aware what the MAC Address of each of the Pi was beforehand, so the UI was able to show it to me easily.

Node Roles

Each Raspberry Pi 4 has a function to perform, we split them into 2 major roles.

Control Nodes

As described in the figure, I will pick 3 Pis as Control nodes. They will be able to control
the worker nodes and also be a bit more fault-tolerant as compared to having just one node control the whole cluster.

Out of these 3 nodes, I will pick on to be the entry-point and main node that I will use to work with and provision my cluster (control01 Pi with 192.168.0.108)

Worker Nodes

The are the real MVPs here. These nodes will do all the heavy lifting for us.

Initial Setup

Download the Ubuntu Server 20.04 LTS 64-bit OS Image
Follow the instructions of burning the image on to the SD Cards
Setup your Pis by inserting the SD Cards, connecting the Pis to the Switch and powering them up (Note: make sure to provide a minimum of 3A current to the Pis or else you might face brown-outs)
Connect the WLAN Router to the Switch and use the router's UI to provide fixed/static IPs to each of the Pis
Connect to the WLAN network and, login into the Pi with address 192.168.0.108 using ssh
```
ssh -l ubuntu 192.168.0.108
```

Note: all the Pis will have the username ubuntu

Hard Part

Be a nice developer / administrator for once and log into each Pi and change the passwords to something safe. Just this one time.

As it is, Ubuntu will insist you on changing the password for the ubuntu user on the very first SSH Login. Might as well begin with good security practices!

For the sake of simplicity, you can keep the same password for each Pi (but preferably you would use some form of password generator for distinct passwords for each Pi)

I promise, this is the only time you will have to login these Pis individually!

You can change the passwords for the user ubuntu and root using:

echo "changing root password"
sudo passwd

echo "changing user Ubuntu's password"
passwd

Setting Up the Cluster

on the Pi with 192.168.0.108 you are currently logged into change the /etc/hosts file:

sudo nano /etc/hosts

We map the names of the Pis to their respective IP addresses and use the .local mdns FQDN for ease of access

127.0.0.1 localhost # This is me, hello!


## This can be adapted according to Your Needs!!
## Cluster Lab Hosts
### Control Nodes
192.168.0.108 control01 control01.local
192.168.0.107 control02 control02.local
192.168.0.106 control03 control03.local

### Worker Nodes
192.168.0.101 worker01 worker01.local
192.168.0.102 worker02 worker02.local
192.168.0.103 worker03 worker03.local
192.168.0.104 worker04 worker04.local
192.168.0.105 worker05 worker05.local

Checking Connectivity

we can use nmap on the pi to ping the sub-network and see if all the Pis are reachable

$ sudo apt install nmap

$ nmap -sP 192.168.0.100-120 # the IP range for the network

This should yield:

Starting Nmap 7.80 ( https://nmap.org ) at 2021-05-05 14:44 UTC
Nmap scan report for worker01 (192.168.0.101)
Host is up (0.00073s latency).
Nmap scan report for worker02 (192.168.0.102)
Host is up (0.00060s latency).
Nmap scan report for worker03 (192.168.0.103)
Host is up (0.00051s latency).
Nmap scan report for worker04 (192.168.0.104)
Host is up (0.00046s latency).
Nmap scan report for worker05 (192.168.0.105)
Host is up (0.00041s latency).
Nmap scan report for control03 (192.168.0.106)
Host is up (0.00034s latency).
Nmap scan report for control02 (192.168.0.107)
Host is up (0.00029s latency).
Nmap scan report for control01 (192.168.0.108)
Host is up (0.00024s latency).
Nmap done: 21 IP addresses (8 hosts up) scanned in 1.41 seconds

Looks good and well-connected!

Introduce Ansible to your Pi Fleet

First and foremost install ansible on the Pi with Address 192.168.0.108

sudo apt install -y ansible

Change the Hosts file for ansible as follows:

sudo nano /etc/ansible/hosts

# Cluster Lab Settings

[control]
## all the "control" nodes go here
# keep control01 as local but other nodes can be reached via ssh
control01  ansible_connection=local  var_hostname=control01
control02  ansible_connection=ssh    var_hostname=control02
control03  ansible_connection=ssh    var_hostname=control03

[workers]
## all the "worker" nodes go here
worker01  ansible_connection=ssh  var_hostname=worker01
worker02  ansible_connection=ssh  var_hostname=worker02
worker03  ansible_connection=ssh  var_hostname=worker03
worker04  ansible_connection=ssh  var_hostname=worker04
worker05  ansible_connection=ssh  var_hostname=worker05

## The complete group of pis here is called a "cluster"
## which comprises of "control" and "workers"
## You can however name the group however you like!
[cluster:children]
control
workers

Remember that at this point of time we haven't even changed the names of the hosts on each Pis,
we have just mapped the IP Addresses to a name! But we will automate such changes via ansible

Passwordless logins via SSH

In the configuration for the ansible hosts, you could see ansible_connection=ssh in all but one nodes. This is achieved by generating an SSH key from the present Pi and shared to all other pis

Okay! I lied before about just the one time doing the redundant task on all Pis. Here you might have to share the SSH Public Key to all other Pis One Last Time!

$ # On cluster01
$ mkdir -p ~/.ssh && chmod 700 ~/.ssh
$ ssh-keygen -t rsa
$
$ # Copy the Key to each node in the cluster
$ ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu@control02 # control03, worker01-worker05 ...

Connectivity via `ansible`

Let me show you why I am blow away by ansible

Ping all the nodes using:

$ # cluster is the name of the group mentioned in the `/etc/ansible/hosts` file
$ ansible cluster -m ping # here -m is module which is <ping>

And surely you get pings from all the nodes thanks to ansible

You can ping just the control nodes using

$ ansible control -m ping
$ ansible workers -m ping # say hello to the Workers

Change the `hostname` for each Pi via `ansible`

Now, remember each Pi on the network is still called ubuntu@ubuntu where ubuntu is also the hostname. To do it manually would comprise of executing the following command in each Pi

hostnamectl set-hostname <desired_hostname_for_pi>

in the /etc/ansible/hosts file we declared var_hostname=... which will set the names up in a single command line instruction

$ ansible cluster --become -m shell -a "hostnamectl set-hostname {{ var_hostname }}"

And Voila! this will be done for all Pis. To check the name changes execute:

$ ansible cluster --become -m shell -a "hostnamectl status | get hostname"

and all the Pis should have the designated hostnames

Changing Configuration for Containers

you need cgroup enabled in the Pi to do fun stuff with containers (docker, kaniko etc.) in the
/boot/firmware/cmdline.txt file.

Do this via ansible:

$ ansible cluster --become -m shell -a \
"sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/firmware/cmdline.txt"

Install `microk8s` on the cluster

You can install microk8s on each pi using Snap from Canonical using:

$ ansible cluster --become -m snap -a "name=microk8s classic=true channel=1.21"

The command will install microk8s v1.21/stable on each Pi

To check the status using ansible:

$ ansible cluster --become -m shell -a "snap list | grep microk8s"

Final Reboot

Since you have made changes if might be better to reboot the Pis you can do this based on the subgroups

$ ansible control02 --become -m reboot
$ ansible control03 --become -m reboot
$ ansible workers --become -m reboot

Finally reboot the current Pi cluster01

sudo reboot

Concluding Remarks

I am honestly blown away by ansible as tool and planning to learn it and other aspects like the Playbooks where you can configure the fleet using files!!

DEV Community: Shan Desai

Exploring the Marshland of Docker Compose Versions with single quotes, special characters and environment variables

Problem

Compose V2: what changes?

Swiss Knife for the Exploration

Vagrant

Ansible

Why are we doing this?

Node-RED Container

Code

Vagrantfile

Ansible Playbook: docker-compose-playbook.yml

Files for reproduction of incompatibility

docker-compose.yml

.env.node-red

Compatibility Checks

Checks: Password not encloses in Single Quotes

Compose v1.25.4 (without single quotes)

Compose v1.29.2 (without single quotes)

Compose v2 without single quotes)

Checks: Password enclosed in single quotes

Compose v1.25.4 (with single quotes)

Compose v1.29.2 (with single quotes)

Compose v2 (with single quotes)

Result: Compose Compatibility Matrix

What did we learn today, kids?

Github Gist

Multi-Image Docker Images: Using COPY with Images directly from registries

Problem

Thousand Leagues under the Container Sea!

Side-Mission

Build everything from scratch?

Charting towards a Destination

Where art thou SDKs?

Multi-Image Image

Solution

Use htpasswd to secure your node-RED Container

Securing Node-RED

Automation: Encrypted Passwords

Automation Script via htpasswd

Jumping right into it!

Creating an Environment Variable Template file for node-RED Docker Container

Creating a settings.js file for Node-RED Container

Creating a docker-compose.yml file

Problem

Investigations

Solution

Resources

Set the hostname of a Docker Container same as that of your host machine

Setting Container Hostname to be same as your Host Machine

Scenario

Ubuntu's mysterious HOST and HOSTNAME environment variables

Investigation via a simple Example

Solution

No Setup Development: Productivity Experience with Docker

Why I stopped worrying about setting up environments!

Problem: So much to download, and setup before getting to work

Solution: Docker encapsulated Environment

Setting it Up!

Caveats

Docker CLI

Benefits

Create a custom Node-Red Docker Image and performance Integration Tests and CI/CD Pipelines

Pre-Requisites

Requirements

Follow Along

Custom Node-Red Docker Container Creation

Adding Node-Red and its Dependencies

Multi-Stage Dockerfile for a Slim Image

Add a Healthcheck API in Node-Red

Integration Test of your Node-Red Image

Testing Files

Additional: using this setup in a CI/CD Pipeline using GitHub

Inference

Prepare your Docker-Compose Stack as a Tarball for Offline Installations

Requirements

Scenario

Usage Steps

Unpack your Tarball and Test

Use Docker Secret to set your Portainer Admin User's Password

`Vagrantfile`

Ansible Playbook: `docker-compose-playbook.yml`

`docker-compose.yml`

`.env.node-red`

Automation Script via `htpasswd`

Creating a `settings.js` file for Node-RED Container

Creating a `docker-compose.yml` file

Ubuntu's mysterious `HOST` and `HOSTNAME` environment variables

Connectivity via `ansible`

Change the `hostname` for each Pi via `ansible`

Install `microk8s` on the cluster