DEV Community: Alex Hyett

Privacy Focused Blog Services

Alex Hyett — Fri, 28 Jan 2022 11:51:41 +0000

Towards the end of 2021, I started getting into web3 a lot more. Not only did I join the Developer DAO but I also launched my own NFT collection and wrote a few blog posts on the topic.

One of the key components of web3 is the ownership of data. In web 2.0, our data is owned by the likes of Facebook, Google and Apple. We have no control over our own data and in most cases, these companies are profiting from it.

Now my blog is a static website created using Gatsby.js. The lack of an underlying database has meant I have had to rely on 3rd party services for analytics and comments.

Until recently, I have been using Google Analytics for tracking the number of visitors and Disqus for my commenting system. However, both of these companies track their users and are profiting from the data they collect. As a web3 advocate, I felt a bit hypocritical writing about all the benefits but still using these services that are tracking data.

So at the end of 2021, I switched to privacy-focused services that don’t collect identifiable information on visitors.

Plausible Analytics

I don’t know the exact statistic but I imagine the majority of the web has Google Analytics running on it. As a result, Google is able to collect an enormous amount of data on the websites you visit and can track your movements around the web.

Obviously, as a simple user, I am not able to identify individual people on my site but Google can, especially if you have a Google account too.

Google Analytics has a dizzying number of features but ultimately there are only a few data points I care about:

How many people are visiting my site?
What posts are they reading?
Where did they come from?

Possibly in the future when I have products I might want to know how many of my visitors convert into paid users.

Plausible Analytics does all of this without compromising the privacy of the users of my site.

Now Plausible Analytics isn’t free at £9 a month but not only are you protecting your user’s privacy you are also helping out an indie development team. Plausible is developed by a small 2 man team Uku Täht and Marko Saric and it is nice to be able to support them rather hand more money over to Google. Plausible also give you the first 2 months free so you can see whether it is the right tool for you.

I also much prefer the dark dashboard compared to Google Analytics:

I have been really impressed with Plausible. I wish they had a mobile app and widgets but the dashboard is mobile friendly so it isn’t a big loss.

Plausible is simple to set up, it is just a case of adding the javascript to your site. It is also a smaller script than Google Analytics so it will help your website performance as well.

I always found it ironic that Google’s Page Speed was always asking me to fix the Google Analytics script due to its size.

Giscus

When I switched from WordPress to Gatsby, I needed to add a third-party comment system. Disqus is very easy to set up but requires users to sign in and will often try and add adverts if you don’t turn them off. I am sure there is a fair amount of data collection that goes on as well.

The downside of comment systems is they cannot be completely anonymous as otherwise, you will end up with a lot of spam on your website.

When I was hunting around for alternatives I found utterances. Utterances is open-source and uses GitHub to store your comments. It seemed perfect but I wasn’t keen on the fact that conversations were being stored as issues.

I then found Giscus a fork of utternances that uses GitHub discussions for the storage mechanism. If you look at the sites you will see they are basically identical in how they work.

You still need a GitHub account to be able to comment but as a developer-focused site, it is likely the majority of my users probably would have one anyway.

Issues with setup

I did hit a few roadblocks when setting up Giscus. The main issue was with the way I had my website set up.

I currently have my website hosted on AWS with my static website hosted in an S3 bucket and CloudFront as my CDN.

When I first tried to sign in to GitHub to comment I would get redirected back to my website but it will still tell me I needed to sign in. The issue was there was a redirect happening which was causing the query string containing the token to get dropped.

First I made sure I enabled query string forwarding in my CloudFront distribution. That didn’t help by itself but I guessed was needed anyway.

In an S3 website if you visit a page hosted in a folder without specifying the trailing slash it will redirect you to the trailing slash version. When it does this, it drops the query string. To me, this seems like a bug with the way S3 hosted sites work, either way, we need to work around it.

The trick is to add a function to your CloudFront distribution that will add the trailing slash for you before it passed to the S3 bucket.

To do this, you go to the Behaviour section of your CloudFront distribution. I have a default behaviour added already to redirect HTTP to HTTPS so I just added to this. If you scroll to the bottom of your behaviour you will find a section called Function association.

Your lambda needs to be hosted in us-east-1 to be able to use it with CloudFront.

In my case, I only need to add a slash if it wasn’t there. This is what my code looks like:

const path = require('path');
exports.handler = async (event) => {
  const { request } = event.Records[0].cf;
  const { uri } = request;
  const extension = path.extname(uri);
  if (extension && extension.length > 0) {
    return request;
  }
  const last_character = uri.slice(-1);
  if (last_character === "/") {
    return request;
  }
  const newUri = `${uri}/`;
  console.log(`Rewriting ${uri} to ${newUri}...`);
  request.uri = newUri;
  return request;
};

You can test your lambda in the console before publishing and hooking up to your CloudFront distribution.

A big thank you to Sean Devine for his post on this, Add Trailing Slash to AWS CloudFront Request

What is DeFi? Understanding Decentralized Finance

Alex Hyett — Fri, 19 Nov 2021 11:52:22 +0000

If you are new to the crypto space the concepts of decentralised finance can seem a bit overwhelming. There are so many new terms to try and understand, it can be challenging to see how everything fits together.

I have been working in the traditional FinTech space for the last 10 years and it has taken me a while to get my head around everything and I am still learning, so you are not alone here.

We are still very early in the web3 crypto space, and many implementations are still quite technical. We are still quite a way off to having mainstream adoption by the general public.

What is Centralized Finance (CeFi)?

Before we try and explain DeFi it helps to understand the system we currently know and tolerate.

Centralised finance encompasses the entire banking system as we know it today.

There are quite a few financial services that have been developed over the last century.

Transferring

This is a pretty basic service. All banks offer the ability to transfer money from one bank account to another. Over the years we have come up with a few ways of doing this:

Internal Transfers - transferring between your accounts in the same bank. These transfers are generally instant.
Faster payments (FP) - transferring money between different supported institutions. This is supported by most banks in the UK (in the US the equivalent is probably ACH or RTP). Generally, transactions are instant but can take up to 2 hours to appear in the target bank account.
CHAPS/Swift - for larger payments that can’t go by FP we have CHAPS. There is a fee to transfer by CHAPS and transfers can take 3 - 5 days.
Transfer Services - On top of traditional transfer services we also have services such as PayPal, Wise, Western Union to name but a few.

Exchanging

If you have ever been abroad you will likely have transferred money into a different currency. Simply converting from one currency to another is one factor the other is trading currencies like stocks and for that we have Forex.

Saving

A savings account is usually one of the first accounts you open when you are younger. Put money in a savings account and as a reward for leaving it there, you earn a bit of interest (like a tiny bit).

If you are willing to lock up your money for longer there are also savings bonds where you get a guaranteed fixed interest rate but you can’t access your money for a set period.

Lending and Borrowing

If you need more money than you currently have in savings then you need a loan. A lot of people have loans for things like houses, cars etc.

To be able to get a loan you need to have a good credit score so they can work out how likely you are to pay the money back.

If you need a loan you used to have to go to a bank. However, there are now quite a few loan services offering “payday loans” usually with extremely high-interest rates.

On the other side, there are also services for lending your money and getting interest in return.

Of course, we also have credit cards which are a form of flexible loan that you only pay interest on if you don’t pay back the money on time each month.

Insurance

Where there is money and risk there is insurance — Me (just now)

In life, there is always a risk. Your house could burn down, you could get into a car accident or someone could steal your stuff. For that we have insurance.

There are thousands of insurance companies that weigh up the risks and decide how much to charge you for peace of mind. In all cases, it is up to the insurance company to decide whether they are going to pay out or not.

Margin trading

We are now getting into the more complex sides of CeFi which you generally have to be into finance to get involved with. So I will explain what these are in case you aren’t familiar.

If you invest £1,000 in stock and the stock goes up 5% you have just made £50. If only you had more money to invest!

Let’s say you borrowed £100,000 instead in this case you would have made £5,000. You then pay back the £100,000 and keep the £5,000. This is what margin trading is.

Of course, it can go the other way, say the stock drops 5% you now only have £95,000 but still owe £100,000.

I am no expert in margin trading so feel free to correct me in the comments :).

Derivatives

Derivatives are a bit like betting. You are betting whether an asset is going to go up (long) or down (short). That is how people make money from stocks going down.

A derivative is essentially a contract between 2 parties to agree to buy or sell at a set price at some point in the future. How much a derivative is worth depends on the value of the underlying asset.

Say for example you want to sell the popular bike you own in 1 months time. You agree to sell your bike to Bob for $1,000 and a contract is written up. In the meantime, the company stops making your bike and everyone is disappointed. The remaining supply of that model is now going for $1,500 in shops.

Now Bob who agreed to buy your bike no longer wants it but knows Alice does. Bob decides to sell the contract to buy your bike to Alice for $100. As a result, Bob has made $100, Alice has saved $400 on that model of bike and you get your $1000 as agreed.

The contract that Bob sold to Alice is a derivative. If Alice wanted to she could have sold it on to someone else before the contract was settled.

There are different types of derivatives such as forward contracts, futures contracts, options contracts and swaps.

In these cases, you are buying the contract and are taking on the profit and loss of the underlying asset without actually owning the asset.

Again, I am not an expert in derivatives but hopefully, that should give you some basic understanding. Feel free to correct me in the comments.

So what is decentralized finance (DeFi)?

Simply, DeFi is creating everything we have in CeFi mentioned above but in a decentralised way.

DeFi solves some of the key issues we have with the centralised system.

Centralised - the current systems are run by individual companies. Generally, you have to give them your money to be able to do anything with it. The company could go bust, be fraudulent or get censored.
Closed source - a company can do anything with your money once they have it. No one outside the company knows how the process works and there is no way to validate things are going to work.
Expensive - Now currently DeFi can be expensive too (e.g. gas fees) but this is only on certain blockchains. It is generally cheaper to send cryptocurrency than it is to send fiat currency.
Identity/Permission - to do anything with CeFi you generally need to hand over your entire identity in the form of ID cards, Passports, Birth certificates etc. Do you trust all that personal information with every company you interact with?

Cryptocurrency

When I think of cryptocurrency I generally compare it to physical notes (cash) rather than money in a bank account. I can give you a £10 note and you then have that money without having to go through someone else.

Cryptocurrency is internet cash. Keep this in mind while we take a look at the services that make up DeFi.

Transferring

This forms the heart of cryptocurrency. With all of the different cryptocurrencies, there is always the ability to transfer crypto from one wallet to another with no middleman.

The CeFi alternative is to deposit cash into a bank account, which is a pot of money owned by a company. Which then gets transferred to another bank account which is again owned by a different company. To then withdraw as cash.

With cryptocurrency, the money goes from me to you without anyone else getting involved or having to hand over your personal details in the process.

Exchanging

There are quite a few crypto exchanges now but not all of them are created equally.

You are probably familiar with the main ones such as Coinbase, Binance and Crypto.com. We need these services to get started in Crypto and convert your fiat currency into cryptocurrency.

However, they are not decentralised. For one, you need to hand over your identity to be able to start buying crypto with fiat. In most cases, your money isn’t sitting in a secure wallet you own. As such it could get hacked or the company could go bust and you would lose your money.

Always move your money out of these services into a wallet you own. Such as a MetaMask wallet or better yet a hardware wallet such as Ledger. I compare MetaMask to the money I keep on my person to allow me to buy things and my hardware wallet as my savings account.

A purely decentralised exchange allows you to move money from one currency to another and is controlled by smart contracts not by an individual company.

Good examples of decentralised exchanges are:

Uniswap - simple exchange with quite a few supported currencies.
SushiSwap - a nice interface for doing exchanges
Kyber - with Kyber you can choose to either lower fees or maximise your return. In some cases, it will even use Uniswap for the exchange if it is preferable.
DEX.AG - DEX is an aggregator and will help find the best exchange to use.

There are also a few non-decentralised exchanges that work on an order based system such as:

Saving

There isn’t a direct comparison with traditional savings accounts in DeFi (feel free to correct me if I am wrong).

The way traditional savings accounts work is a bank acts as a middleman between lenders and borrowers. Banks will give you >1% interest on your savings which they are then lending out to others with an interest rate of 2 - 19% depending on the type of loan.

You get a higher interest rate on bonds because banks know exactly how long they have your money for and how long they can loan it out.

DeFi is all about removing the middleman as such we will go straight to the next section on lending and borrowing.

Lending and Borrowing

The first lending protocol that was introduced was the MakerDAO smart contracts. MakerDAO introduced DAI as a stablecoin. DAI is pegged to 1 USD, this is done programmatically through the use of smart contracts.

In order to get DAI, you need to deposit another cryptocurrency to use as collateral. In fact, you need to deposit 150% of what you want to borrow.

So why would you want to do this?

Let’s say you want to buy another coin like BAND. So you swap your 1 ETH (currently at ~$4,600) for the equivalent in BAND.

BAND then drops 50%. You now only have $2,300 worth of BAND. In the meantime, ETH has skyrocketed to $8,000 and now you really wish you had kept it in ETH.

Now, let’s say instead you deposited it for DAI. You now have $3,000 worth of DAI (remember you need 150% as collateral).

You do the same thing and buy BAND. It again drops 50% as before. However, your original ETH is still tied up in your loan. If you can get $3,000 DAI plus the interest you can get back your 1 ETH which is now worth $8000.

There are now lots of other decentralised lending platforms that let you lock up cryptocurrency and get back interest in return. Alternatively, you can use it as collateral for taking out a loan. Smart contracts ensure that your collateral doesn’t drop below the loan amount.

The most popular lending platforms are Compound and Aave.

If you deposit ETH you will get back cETH with Compound or aETH with Aave. These tokens are pegged against the original amount you deposited and will earn interest every second you hold them.

Insurance

Even with everything being decentralised there are still risks with crypto. For example, your exchange could get hacked or there could be a bug in the smart contract you are using.

Buying “insurance” works a bit differently from traditional insurance companies. These decentralised insurance providers are owned and run by their members and use Ethereum smart contracts.

With Nexus Mutual members buy NXM and then stake them against smart contracts that they believe are secure. The members then receive rewards when users buy insurance for those contracts and don’t claim on them.

Margin Trading

Margin trading is when you borrow money to invest so that you get larger returns. Being decentralised you borrow money from lenders who in return earn interest from you on the money borrowed.

Everything is controlled using Smart Contracts. Borrowing money to buy another asset is called a leveraged position. These leveraged positions are controlled by smart contracts. If the value of your asset you bought drops too much then the leveraged position is automatically liquidated.

Some of these exchanges also have additional insurance in place so that lenders don’t lose their initial investment.

Fulcrum
dydx

Derivatives

Derivatives work in a similar way as in the CeFi world but the derivatives are managed using smart contracts.

Derivatives in DeFi aren’t just limited to crypto though. You can buy derivatives in the same things you can in the CeFi world. However, the key difference is that derivatives are just ERC-20 tokens and as such can be exchanged for other tokens on DeFi platforms or invested again.

Synthetix

Final Thoughts

Hopefully, this post gives you a basic understanding of what CeFi and DeFi are and how they differ.

I am not a financial expert so feel free to correct me if something isn’t right. If you know of any other platforms that are worth mentioning in this article let me know in the comments and I will add them.

Using Hashnode as a Source for Gatsby

Alex Hyett — Fri, 05 Nov 2021 17:51:49 +0000

I have done quite a few website migrations over the years. My blog like many first started out on WordPress. After several attempts at optimisation, I ended up generating a static version of my WordPress website.

On my static site generation journey, I discovered Gatsby.js and completely redesigned my website around it. However, writing posts locally in markdown has its own downsides and lacks the discoverability you get from blogging on Hashnode.

So I decided to move my blog posts over to Hashnode but I still wanted my posts to appear on my main blog.

Migrating the Blog Posts

The first step was to import all my blog posts to Hashnode. There are a few methods you can use to import your blog posts.

I decided to use the Dev.to importer as I repost all my posts on dev.to anyway. Unfortunately, the bulk import didn’t work for me so I had to import my posts one at a time.

This was handy though as you need to set the following for each post manually:

Post slug
Tags
Canonical URL

Once all my posts were imported it was time to edit my Gatsby website to start pulling down the posts.

Gatsby plugins

If you do a search for gatsby-source-hashnode you will find a couple of plugins on the Gatsby website.

I decided to go with the first one (gatsby-source-hashnode) as it has a lot more downloads and still appears to be maintained.

However, I quickly noticed when testing out the plugin that it was only pulling down the first 6 posts from my Hashnode blog. This might be a recent change to the API which hasn’t yet been picked up by the plugins.

Luckily, the plugin is open source so I raised a PR for the plugin owner to review.

In the meantime, I copied my modified code to a folder in my gatsby repository called plugins/gatsby-source-hashnode. I then used the plugin as described in the documentation.

GraphQL queries

I am using GraphQL queries in a few places to pull down posts:

Blog Feed
Blog Post
Latest Posts
RSS Feed

For reference, this is what a couple of my queries look like for pulling from Hashnode.

Blog Feed

query pageQuery($skip: Int!, $limit: Int!) {
    site {
      siteMetadata {
        title
        description
      }
    }
    allHashNodePost(
      sort: { fields: [dateAdded], order: DESC }
      limit: $limit
      skip: $skip
    ) {
      edges {
        node {
          brief
          slug
          title
          coverImage {
            childImageSharp {
              gatsbyImageData(
                width: 920
                height: 483
                layout: CONSTRAINED
                transformOptions: {cropFocus: CENTER}
              )
            }
          }
        }
      }
    }
  }

Blog Post

query BlogPostBySlug($slug: String!) {
    site {
      siteMetadata {
        title
        author
        siteUrl
      }
    }
    hashNodePost(slug: { eq: $slug }) {
      _id
      brief
      childMarkdownRemark {
        htmlAst
      }
      slug
      title
      dateAdded
      coverImage {
        publicURL
        childImageSharp {
          gatsbyImageData(
            width: 920
            height: 483
            layout: CONSTRAINED
            transformOptions: {cropFocus: CENTER}
          )
        }
      }
    }
  }

I am not yet pulling down tags or reading time from the API. That is something I will try out later.

Automation with GitHub actions

I already have GitHub actions set up for my website so that any new commits to the main branch trigger a website build and push to S3. I wrote a post about it if you want to find out how.

However, now that I am not committing articles to that repo I needed a way to automatically trigger builds when I publish a new post on Hashnode.

Luckily, Hashnode has an option to backup your blog posts to GitHub which we can use as the trigger point.

Step 1: Add workflow_dispatch to your GitHub action

To be able to remotely trigger a build on GitHub you need to add workflow_dispatch to the on: section of your workflow file.

This is what the top of mine looks like:

name: Deploy Blog

on:
  workflow_dispatch:
  push:
    branches:
      - main

Step 2: Create a personal access token

Next, we need to create a personal access token that we will use as the API Key for triggering the remote build.

You can do this in GitHub under Settings > Developer Settings > Personal Access Tokens

I set my access token with the following permissions.

You will need to set an expiry date as well and set yourself a reminder to generate a new token before it expires.

Make sure you copy the access key as you won’t get a chance again!

Step 3: Add the key as a secret to your backup repository

To be able to use this access key we need to add it as a secret to the repository where Hashnode is backing up your posts.

On the backup repository go to Settings > Secrets and add a new repository secret called ACCESS_TOKEN and put in the key from the previous step.

Step 4: Add a workflow file

Next, we need to add a workflow that gets triggered every time a commit is added to this repository. In our case, this will be whenever we publish a post on Hashnode (Note: Drafts aren’t saved in GitHub).

Add the following file to your backup repository .github/workflows/workflow.yml

--------
name: Trigger Deploy

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - run: |
          curl -X POST \
          -H "Authorization: Bearer ${{secrets.ACCESS_TOKEN}}" \
          -H "Accept: application/vnd.github.v3+json" \
          https://api.github.com/repos/GITHUB_USERNAME/WEBSITE_REPO/actions/workflows/workflow.yml/dispatches \
          -d '{"ref": "main"}'

Make sure you change GITHUB_USERNAME to match your GitHub username, WEBSITE_REPO to the repository with your Gatsby.js website and workflow.yml to match the name of your deploy workflow file.

To test, try updating one of your posts in Hashnode and you see this workflow get triggered then followed by your website workflow.

What is Web3? The Decentralised Internet Explained

Alex Hyett — Fri, 15 Oct 2021 12:00:00 +0000

If you have stumbled upon this article you have probably seen the term web3 being mentioned around the internet. Especially on Twitter at the moment it seems to be the latest buzzword.

With all the hype around NFTs and crypto at the moment, a lot of people are assuming that this is all web3 is. However, web3 (or web 3.0) has a lot more to it than that.

Before I go into more details, let us look at how we got here and what we consider web 1.0 and 2.0.

Web 1.0: The Read-Only Web

When the internet first went mainstream it consisted of static HTML pages that were uploaded to servers using FTP.

People who had the technical skills were the only ones that could contribute to the internet at that time. The internet was limited to news sites and technical sites and wasn’t of great interest to the general public.

This stage of the internet is now nicknamed the read-only web.

Web 2.0: The Social Web

Soon websites started to be backed by databases instead of static files. It was now possible to interact with websites and not just read them.

Websites started allowing people to comment on their articles and soon we had online stores that we could buy from.

As traffic to these websites increased people started looking for ways to monetise their work. As with traditional media such as magazines and newspapers, the solution was to sell advertising space on their website to companies looking to advertise their products and services.

In the early days of Web 2.0, apart from comments the content on the web was still mostly produced by website owners.

Social media begins

In 2003 MySpace was launched. People now had a space to create their own content and share it with others. By 2005 MySpace was seeing 16 million monthly users and was acquired by News Corporation for $580 million.

Facebook was already out but limited to university students. This all changed in 2006 when Facebook opened up to everyone over 13. It didn’t take long for everyone and their Mum to have a Facebook account and sharing everything about themselves with the world.

With everyone using Facebook and Google, these companies amassed a large amount of data about their users.

Facebook was looking for ways of monetising its site. As selling the user data was morally wrong (although it didn’t stop some) they instead went with allowing people to target users with adverts.

Instead of paying for advertising space on a website and hoping your target market was looking, you could now push adverts directly into people’s Facebook feeds based on age, sex, location and interests.

Social media companies started reaching billion-dollar valuations even before they started monetising due to the information they had on their millions of users.

If the product is free, you are the product.

Tech companies didn’t need to sell products or charge for the use of their service. Instead, they could collect data on their users and sell access to that anonymised data to companies.

The users became the product.

What is wrong with the internet now?

Of course, with there being so much value in user data these companies became targets for hackers and over the last decade, there have been a lot of high profile data leaks.

Not convinced, sites like Have I Been Pwned can show you whether your details have been compromised. There is also a very long list of data breaches listed on Wikipedia.

If you have used any of the major websites chances are your details have been leaked.

In some cases, it hasn’t been the result of hackers but the companies outright selling your data.

Where is your data?

If you are an active internet user like me, chances are you have lost track of the number of websites that you have given your email address and other information.

There are so many services that I have signed up to but later decided it wasn’t what I was looking for. Most of these sites don’t have ways to delete your account and there is no way of knowing they do delete your details anyway.

Let’s take the example of buying something from a trusted website.

As part of the checkout process you have to enter your name, email, card number and billing address (even if you are buying a digital product). These are stored (hopefully securely) on the servers of the company you are buying from. Depending on how the company has set up their store this could be another third party like Shopify.
To be able to take payment for your order, they need to use a payment gateway like Stripe. So your name, email address, card number and billing address are now on the payment gateway’s servers too.
Did you get an email confirmation of your order and possibly some marketing emails? Well, your data is now stored on their email providers servers such as Mailchimp.
If you bought a physical product then I can guarantee the company isn’t going to hand-deliver it to you. They are going to use a courier and as such your details would be given to them as well so they can deliver your product and contact you in case of difficulties.

Chances are you would have agreed to all this in the 20 pages of terms and conditions you agreed to but didn’t read.

Of course, to be able to have a credit or debit card you would have had to hand over all your details and money to a bank. They also would have used a bunch of third parties to be able to validate who you are and print the cards you are using.

Web 3.0: The Decentralised Web

In 2008 a whitepaper was written by someone called Satoshi Nakamoto on a peer to peer cash system that lives on a “blockchain” and Bitcoin was born.

With cryptocurrency, it is now possible to send money (or crypto tokens) from one wallet to another without sharing any personal information. No third party is involved it is purely peer-to-peer.

Now if this blockchain was hosted by a single company it wouldn’t be much better than the payment system we have now. However, the blockchain is decentralised, not owned by a single entity and impossible to take down. All the payments are immutable and verified by the network.

If something is completely decentralised then it cannot be taken down as it doesn’t exist in a single place. Of course, this is useful for more than just payments.

The power of web3 is the ability to be able to create applications and websites that live and work with the blockchain. Completely decentralised and available to everyone.

NFTs

NFTs or Non-Fungible Tokens are assets that can be bought for cryptocurrency and stored in your crypto wallet.

You will probably be familiar with NFTs from people talking about NFT art. NFT Art is about ownership. Yes, it might just be a JPEG that you are buying but you own it and can prove that you own it.

It is the digital equivalent of owning the Mona Lisa versus owning a print.

The uses of NFT go far beyond the artwork. It provides a trustless system for ownership. Anything digital that you own can be represented as an NFT and then verified by a third party that you are the owner.

Nearly every website you log into requires a username and password. With web3 all you need is your wallet connected with your NFT. No personal information needs to be transferred.

Not only that but you can transfer your ownership to someone else. This is something that the digital space has been sorely lacking.

Let’s have a look at a few examples:

Digital Media

At some point, you have probably bought a digital copy of a book, game, song or movie.

Chances are it wasn’t much cheaper than buying the physical object. However, with a physical book, you can donate or sell it once you are done. With a digital copy, you don’t have that option.

However, if that book or game was an NFT in your wallet you would be able to donate or sell it when you no longer wanted it.

Authorisation

An NFT can also be used to provide access to a resource. By having the NFT in your wallet a web3 website can check whether it exists before granting you access.

Let’s say you bought an online course. You might be given lifetime access to this course but if you find you no longer need it you are stuck with it.

However, if course access was granted via an NFT then you would have the ability to sell or donate that course when you were done with it.

This can be used in the physical world too. Any event that requires a ticket could be sold as an NFT. Only those with that NFT in their wallet have access.

Gaming

I love the scene at the start of Ready Player One where they all fighting in one of the game rooms to try and get the special item (I think it was a sword but I need to watch it again).

In the online gaming world, there are already unofficial markets for gaming items. I never played WoW but I remember people selling their rare items on marketplaces.

The problem is you can easily be scammed as you are trusting the user to hand over the item once you have paid. Or trusting the marketplace to verify that the money and items have been transferred.

If game items were NFTs they would be yours to own and transfer.

There are already some games coming out that rely on NFTs and these are being coined Play-To-Earn games. Axie Infinity is the most popular at the moment.

In some cases, it is possible to earn several thousand dollars a month playing this game.

This game is particularly popular in some countries in Asia at the moment as people can earn more playing this game than they can from local jobs. What would you rather do, work in a factory, on the fields or at home playing a video game?

Unfortunately, you need to own 3 Axies to start playing and with prices starting at 0.034 ETH ($129) per Axie it can be steep for some.

Axie Infinity has started offering sponsorships as an option. Where people can pay for the Axie’s for someone else and in return get a percentage of the profits.

The Spatial Connected Web

The other aspect of web3 is how the physical world is going to merge with our digital one.

We now have a few VR (virtual reality) headsets on the market as well as AR (augmented reality) built into our phones.

Facebook is also currently working on AR glasses, I am sure other companies too.

As more devices get released, there are sure to be more uses for them. The same way everyone has come to embrace wireless payments at some point the same will be true for AR and VR.

You can read more about the spatial web in the book The Spatial Web.

Potential uses

Imagine wearing your AR glasses to the supermarket and being shown a route to the items on your shopping list.

AR could also be used to subtly highlight the products on the shelves that meet your dietary requirements. No need to look at labels to check for wheat or dairy.

Looking online for clothes but not sure how they would look on you? Well, your AR glasses could help with that too. Augmenting the clothes you pick onto your body, maybe even helping you pick the correct size.

Can’t get out at the moment? Wellm why not visit the same shops but virtually and pick the items from your shelves in VR.

The problem with these uses at the moment is it is dependent on device support. It isn’t going to work if some stores only support certain devices.

Imagine being able to use your Apple Glasses in Waitrose but you need Samsung Glasses in Asda. It just won’t work. This is where the metaverse comes in.

Metaverse

The virtual world (or worlds) and the augmented world is being collectively called the metaverse. The metaverse is an interconnected network of worlds that allow continuity of identity and objects.

Currently, the apps and games we use are often dependent on the devices we have.

Some games are only available on XBOX or Playstation while others can be played on all devices.

Even online games available on all platforms have limitations between them. Sometimes you can’t see other players on a different device than yours and you certainly can’t transfer your saved game between different types of devices.

The metaverse is completely connected so not only would you be able to move your identity between different devices but also between different virtual worlds. Everything is interconnected. Say you win a sword in one game, you could then take this sword with you and use it in another game.

Again, I am going to reference Ready Player One here. Where no matter what virtual world they were in, their identity travelled with them, as did their inventory and they could use that in any of the other worlds.

Where do you think your identity and inventory would live? You guessed it, your digital wallet on the blockchain.

With the help of AR you could even see your VR items in the real world too.

DAO: Decentralised Autonomous Organisations

With everything else being decentralised then why not businesses. Instead of a business being owned by a CEO and its founders, a DAO is own by all of its token holders. Each token holder is given voting rights and can decide on the direction the DAO will take.

Any profits made by a DAO are then evenly distributed to the token holders.

There are a few DAOs going around at the moment. It is worth having a read about them and see if you want (or can) join them.

Where we are now

This all sounds great but we are only at the beginning at the moment. We are a little way off chasing keys to the kingdom in the Oasis metaverse just yet.

So we have our decentralised currency that, will work across all our virtual worlds and soon the physical world. Bitcoin is the main candidate and now being legal tender in El Salvador.

However, we also have a mind-boggling array of other cryptocurrencies. Ethereum is the most well known. It builds on Bitcoin by having smart contracts and assets which give us NFTs. It is currently hindered by the expensive “gas” fees that are needed to do anything.

Until cryptocurrency is accepted in physical stores we are still reliant on exchanges to convert the currency back into local fiat currencies. This usually involves handing over a lot of personal information before you can make the conversion.

At some point, we may select a winning cryptocurrency but for now, we need to cope with all the different varieties and rely on exchanges to convert between them.

Web3 websites are starting to crop up here and there mostly around the NFT space. If you own an NFT you gain access to their community and unlock areas of their website. I expect we will see more websites using NFTs for access soon.

Getting started with Web3 Development

Currently, web3 is still browser-based. You will need a browser extension to be able to access web3 features.

You need the Metamask extension, that we allow you to connect your wallet to web3 websites. Provided you have funds in your wallet you will be able to mint NFTs on their websites as well active the owners only features.

If you want to get started with web3 development there are some great resources out there already.

Nader Dabit has a couple of great articles on dev.to. He also has some great content on his YouTube Channel
If you are interested in building NFTs then Daniel Botha has loads of videos on the subject on his Hashlips YouTube channel.
Join the Developer DAO. You will need to mint a “free” (gas fees still apply) NFT token to join https://developerdao.vercel.app/. There are only 8,000 so join soon. This will give you access to contribute to the discord community.

I am still getting started myself with web3 but it is an interesting space and I am sure to post more content here soon.

Ditch the Ads: The Future of Website Monetization

Alex Hyett — Fri, 08 Oct 2021 12:00:00 +0000

Do you remember that scene in Ready Player One, when IOI are showing what the future of the Oasis will look like once they take over?

“We have determined that we’ll be able to fill 80% of the user’s display with advertising before inducing seizures”

Display Advertising

We aren’t living our lives in virtual reality just yet but already it feels like adverts are taking over the web. Especially if you go on some of the big news websites it can be hard to distinguish the content from the adverts.

I recently experimented with using Google Adsense on this website and the results were surprising. Now, this isn’t a high traffic website. I currently only get around 250 - 300 page views a day. However, I was still able to “earn” on average £0.50 a day. This isn’t bad considering I haven’t written a post in 4 months and this is entirely passive income.

Over the month, this would have added up to around £15, which is more than enough to pay for this websites expenses. However, with a minimum Adsense payout of £60, it would have been 4 months until I was able to cash out.

I used the Auto Ads feature of Google Adsense and made sure I turned off the personalised ads and the obnoxious page filling ads. I also made sure to only include ads on blog post pages and turned the ad load down to half of the recommended setting.

This resulted in around 5 - 8 adverts per blog post mixed in amongst the content. If I was only interested in the money, then I could have left them on and in a few more months I would have got a paycheque from the big G.

However, I wasn’t comfortable not knowing exactly what adverts were being shown on my website and with the auto ads feature you have little control over where the ads appear.

Adverts have given website owners a lazy way of generating an income from their content but times are changing and more people are choosing to block adverts.

As a software developer with a tech-savvy readership, the chances of my readers choosing to block adverts is quite high. I even have a Pi-Hole set up in my house to block adverts on all my devices (I should really do a tutorial on this). So it seems a bit hypocritical to serve adverts on my website as well.

So what is the alternative? It turns out there are quite a few alternatives to make an income from your content without resorting to advertising.

Medium

As part of my blog post syndication, I also re-publish my blog posts on Medium with a canonical link back to my website. As you are probably aware, Medium has a paid membership service where you pay $5 a month which allows you to get access to all articles on Medium.

For publishers you get a portion of this membership, depending on how many people read your articles. It is similar to how authors earn revenue on course sites like Pluralsight.

My articles aren’t that popular on Medium. I think part of that is because since they are re-published rather than Medium exclusive articles. So not only are they less likely to be promoted by Medium, they are also less likely to rank in Google as the original article is on alexhyett.com.

So far this year I have earned $0.40 (£0.30) from Medium. I won’t be cracking open the champagne just yet.

However, on Medium, I have only had 26 people read my content according to the Medium stats. So actually $0.015 per reading isn’t that bad. I will continue publishing on Medium and may even do some Medium exclusive posts if the content doesn’t quite match what I normally publish on this site.

Web Monetization

With the rise of cryptocurrency coming into mainstream use, we now have other ways of transferring money from one person to another.

Including the ability to stream fractional amounts of money to the content creators you visit.

Metadata tag monetization

This has now made its way into the browser and is also being proposed as a W3C standard. You can read more about the standard at webmonetization.org.

This is all done with a metadata tag on your website. This is what mine looks like:

<meta name="monetization" content="$ilp.uphold.com/3QmRGLJq4F9d">

You can get your own wallet address by signing up with an account with Uphold or Gatehub. This will allow you to start receiving funds via the Interledger network.

At the moment there is only one Web Monetization Provider called Coil. Coil is a bit like Medium, in that it costs $5 a month and that money is used to distribute funds to the websites you visit. You will need to install the Coil Extension in your browser to start supporting content creators.

Once the above has been added to your website you will start to receive micropayments from your readers who are signed up for Coil. You get $0.36 for every hour a visitor spends on your site. Payments are made using micropayments every second per visitor.

You can even link this to your Dev.to account and receive funds from your readers there too. You will find this option under Settings > Extensions > Web monetization.

If you sign up for Coil and download the extension you will see this when you are on a monetised website:

You can actually see money streaming into your account in the Uphold app which is pretty cool. To be able to transfer the money to your account you need to have received at least £0.02 which isn’t hard.

It is also possible to make exclusive content for paying visitors which is something I will experiment with soon.

Paywall

Talking of exclusive content. There are more and more websites that are locking all of their content behind a paywall. This is in response to dropping ad revenue.

Generally, this is being used by big established media outlets like The Times or The Guardian. However, there are smaller websites that are locking content behind membership sites. They have most of their content available for free but for additional help, exclusive information and access to a community a membership option is available.

If you are expecting people to pay monthly for access to your content you need to make sure you are committed to creating content on a regular basis.

Brave Browser

The other option for streaming money is via the Brave Browser. This browser is based on Chrome but is security and privacy-focused and has better support for Web3 with cryptocurrency wallets built-in.

Brave will block ads and trackers for you while you visit websites. This does mean that sites that rely on adverts for their income won’t be getting any money from your visits. To counteract this Brave have come up with Brave Rewards. With Brave Rewards turned on you will get the occasional notifications adverts that don’t track you or use your data.

Each time you view an advert you earn BAT tokens. These can then be converted to fiat currencies by you or you can use them to contribute to the websites you visit or you can even tip a website directly.

To become a creator and earn BAT from visitors to your site you will need to signup and verify your website.

Once verified visitors will see this in the browser:

Users then have the option to Send a Tip or add in a monthly contribution.

Donations

The other option you may have noticed on my website is my Buy Me A Coffee button. We don’t think twice about spending £3 - 4 on a coffee at Starbucks but we don’t even consider paying for content that we consume. A long blog post can take many hours to research and write and the chances are the blog owner won’t earn anything from that.

If a blog post has helped you and is written by a single developer then sending that person a “coffee” can be a great way to support them.

Buy Me A Coffee is set up to send money directly to Stripe or Paypal which means there is no minimum payout, unlike ad networks.

I am very grateful to the few people that have already bought me a coffee as a thank you for the articles I have written.

Affiliate Links

The other way many website owners earn money is through affiliate links. Affiliates are people who promote another companies product and in return earn a commission.

The person clicking the link doesn’t pay anymore but if they buy something, you will earn a commission from the sale. Lots of websites have affiliate programs you can sign up for. The commissions can vary from 4 - 25% and generally there will be a minimum payout before you can receive the funds.

As a software developer, I don’t talk about a lot of products. The software I use daily is free after all (Visual Studio Code). However, I do have affiliate links to the physical things that make up my home office.

Product Sales

Finally, the last ad-free way of making money on your blog is to sell products. This could be eBooks, software, website templates or courses. Of course, a lot of effort goes into building out a product but it can be very profitable.

I have made some money in the past from website templates and desktop software and it can be a good little side hustle or even a full-time income.

Dealing with Imposter Syndrome as a Software Developer

Alex Hyett — Fri, 28 May 2021 11:30:00 +0000

I have been a professional software developer for over a decade and I have been writing code for over 25 years.

However, sometimes I still feel like a fraud.

It turns out, I am not the only one that feels this way and they have a name for it, it is called “Imposter Syndrome”.

xkcd: Imposter Syndrome

What is Imposter Syndrome?

Have you ever felt like you don’t know what you are doing? Do you feel like you aren’t good enough for your job, that you don’t deserve to be where you are in life?

That everyone else seems to know more than you do and your are just stumbling along working things out as you go along. One of these days they are going to find out you don’t know what you are doing!

If you have, you likely suffered from imposter syndrome.

Imposter Syndrome Definition

Dictionary.com defines this as:

anxiety or self-doubt that results from persistently undervaluing one’s competence and active role in achieving success, while falsely attributing one’s accomplishments to luck or other external forces.

Essentially, imposter syndrome comes about when you doubt your abilities.

It is common in all careers but I think it is especially apparent in the software development industry.

Imposter Syndrome as a Programmer

We are constantly having to keep up with technologies and learn new skills. As such, it is impossible to be able to know everything. There will always be gaps in your knowledge and someone else is bound to know something that you don’t.

Anyone who has worked in frontend development will relate to this. With new frameworks coming out every year, it can feel like you are constantly on a hamster wheel never getting to the finish line. You dedicate your time to learning Angular just to find out that everyone is switching to React or Vue, and now you are back to square one.

This becomes especially apparent when you start to specialise in a particular area. The more time you dedicate to one subject, the less time you are going to have to learn about new things.

This always reminds me of this quote:

You learn more and more about less and less until you know absolutely everything about nothing. - Nicholas Butler

I would consider myself a full stack developer. I know enough technologies to be able to take a product from idea to production including, frontend, backend, database and infrastructure.

However, I specialise in backend development. My frontend knowledge is limited to creating my own personal projects or updating existing projects. I wouldn’t say I was up to date with the latest best practices. My database knowledge is fairly good but I don’t know as much as a dedicated DBA. My infrastructure knowledge is getting better but I am constantly learning from my colleagues.

Dunning-Kruger Effect

What makes matters worse is something called the Dunning-Kruger effect. Another definition, this time from Wikipedia:

The Dunning–Kruger effect is a hypothetical cognitive bias stating that people with low ability at a task overestimate their ability.

With the “Anyone Can Code” movement (which I am not entirely against) we have more and more people who know just enough to be dangerous. In some cases, they have only just started and think because they can put an HTML page together that they suddenly know all there is to know. Or they might have a couples of years experience and have got very good at one thing.

These people tend to have a lot of confidence, as they are unaware at just how clueless they are about everything they don’t know.

Whereas those of us with many years experience under our belt, understand that there is a lot that we don’t know and therefore you shouldn’t get too cocky with what you can do.

However, as we end up feeling inadequate we are less likely to share what we do know with the world.

How to Get Over Imposter Syndrome?

So know that you understand what imposter syndrome is (and probably relate to a lot of it), how do we deal with this?

Recognise your accomplishments

A lot of us tend to downplay our accomplishments. Maybe you have just done a difficult release, managed to solve a bug that has been driving your team mad or even just completing your work for the sprint on time.

If you just move on to the next task without patting yourself on the back it can feel like you aren’t accomplishing anything.

I recommend you write down your achievements. This is especially important when it comes to review time and you need to tell your boss why you should get a pay rise.

Share your knowledge

This is something that I am trying to do more. Push myself out of my comfort zone and get out there and share what I know.

Whether this is via blog, company talk, Instagram, Twitter or YouTube, sharing is an important part of giving back to the development community and helping aspiring engineers improve their skills.

I forget that I have been writing code for 25 years and end up thinking that I haven’t got anything worth sharing. However, not everyone is in the same place you are. Even if you are just starting out there is going to be someone who doesn’t know what you know and would benefit from your knowledge.

So don’t be afraid to get out there and start that blog that you have been putting off for years.

It is always nice to know that you have helped others and it can stop you from feeling like a fraud.

Be Humble

This is my motto for life and the number 1 advice I give to others, “Be Humble”. Once you realise that you do actually know more than other people it can be difficult to keep your ego in check (how do you get your head out of the door?).

So remind yourself to always be humble. Don’t look down on others that don’t know as much as you, as I can guarantee they know more than you about a different topic.

Don’t be afraid to ask questions, even if you think it will make you look stupid. Chances are that if you don’t know the answer there will be someone else who doesn’t either. It is always better to clarify something than assume.

Never ASSUME as it will make an ASS out of U and ME

Final Thoughts

No matter how much you know, there is going to be plenty of things you are clueless about. Instead of getting down about it, take the opportunity to learn more and share your knowledge with others.

Hopefully, this post will be helpful for those going through something similar.

Using WireMock.net for Integration Testing

Alex Hyett — Fri, 21 May 2021 14:40:00 +0000

Last week I showed you how you can use Wiremock in a docker container to mock API calls that your application uses. This week is on a similar theme, but this time using WireMock.net to help with your integration tests.

Unlike WireMock which is Java-based, as you can guess from the name WireMock.Net is written in .NET and based on mock4net. Everything you can do in Wiremock you can also do in WireMock.net. This also includes hosting it in a docker container like we did in the last post (the ASCII art is missing though).

In this tutorial, we are going to be looking at using WireMock.net to mock API calls in our integration tests.

Getting started

First of all, we need a .Net project to test. For this project I am just going to create a new .Net Core Web API from the templates:

dotnet new webapi

This will generate a simple API that gives us a random weather forecast.

Running dotnet run and then calling the GET endpoint https://localhost:5001/WeatherForecast returns the following:

[
  {
    "date": "2021-05-22T10:22:17.824198+01:00",
    "temperatureC": 36,
    "temperatureF": 96,
    "summary": "Chilly"
  },
  {
    "date": "2021-05-23T10:22:17.824551+01:00",
    "temperatureC": 25,
    "temperatureF": 76,
    "summary": "Warm"
  },
  {
    "date": "2021-05-24T10:22:17.824557+01:00",
    "temperatureC": 20,
    "temperatureF": 67,
    "summary": "Hot"
  },
  {
    "date": "2021-05-25T10:22:17.824557+01:00",
    "temperatureC": 14,
    "temperatureF": 57,
    "summary": "Bracing"
  },
  {
    "date": "2021-05-26T10:22:17.824558+01:00",
    "temperatureC": -15,
    "temperatureF": 6,
    "summary": "Sweltering"
  }
]

You can tell it is random, as I wouldn’t consider -15°C, “Sweltering”.

The actual code that generates this looks like this:

[ApiController]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
    private static readonly string[] Summaries = new[]
    {
        "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering",
    };

    private readonly ILogger<WeatherForecastController> _logger;

    public WeatherForecastController(ILogger<WeatherForecastController> logger)
    {
        _logger = logger;
    }

    [HttpGet]
    public IEnumerable<WeatherForecast> Get()
    {
        var rng = new Random();
        return Enumerable.Range(1, 5).Select(index => new WeatherForecast
        {
            Date = DateTime.Now.AddDays(index),
            TemperatureC = rng.Next(-20, 55),
            Summary = Summaries[rng.Next(Summaries.Length)]
        })
        .ToArray();
    }
}

There is one subtle change that has to be made to be able to run integration tests against this. That is to replace the IHostBuilder in Program.cs with an IWebHostBuilder.

Your Program.cs should look like this:

using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;

namespace WireMock.Net.Api
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateWebHostBuilder(args).Build().Run();
        }

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args).UseStartup<Startup>();
    }
}

Adding an API to Test

The whole point of an integration test is to test the integration between components. So we need to add an API to our project so that we can mock it using wiremock.net.

As this API is already returning random weather, we are going to add in real weather API to get back realistic data. For this, we are going to use 7timer’s weather API. This API is free and doesn’t require any authentication.

The API we are going to call is this one:

https://www.7timer.info/bin/civillight.php?lat=51.5074&lon=0.1278&ac=0&unit=metric&output=json&tzshift=0

This returns back weather for the next 7 days for London (51.5074° N, 0.1278° W). The format of the data we get back is slightly different so we will need to format it before returning the data. This is what the first few entries in the array look like:

{
  "product": "civillight",
  "init": "2021052100",
  "dataseries": [
    {
      "date": 20210521,
      "weather": "lightrain",
      "temp2m": { "max": 11, "min": 9 },
      "wind10m_max": 4
    },
    {
      "date": 20210522,
      "weather": "lightrain",
      "temp2m": { "max": 10, "min": 7 },
      "wind10m_max": 3
    },
    {
      "date": 20210523,
      "weather": "lightrain",
      "temp2m": { "max": 12, "min": 4 },
      "wind10m_max": 4
    }
  ]
}

For our API we need the following information:

dataseries[].date - The date which we can just convert to a DateTime.
dataseries[].temp2m - This is the min max temperature in Celsius so we can just use this as is. We can update our API to return both.
dataseries[].weather - The weather summary, again we can use as is.

I am going to use Refit for calling the API. If you haven’t used Refit it is a great tool for standardising your API calls and saves on a lot of boilerplate code for calling your API. So add the following packages to your API project:

<PackageReference Include="Refit" Version="6.0.38"/>
<PackageReference Include="Refit.HttpClientFactory" Version="6.0.38"/>

Next we need to create an interface for our API:

using System.Threading;
using System.Threading.Tasks;

using Refit;

using WireMock.Net.Api.Client.Model;

namespace WireMock.Net.Api.Client
{
    public interface IWeatherClient
    {
        [Get("/bin/civillight.php?lat={latitude}&lon={longitude}&ac=0&unit=metric&output=json&tzshift=0")]
        Task<ApiResponse<WeatherResponse>> GetWeatherAsync(decimal latitude, decimal longitude, CancellationToken ct);
    }
}

My WeatherResponse class looks like this:

using System.Collections.Generic;

using Newtonsoft.Json;

namespace WireMock.Net.Api.Client.Model
{
    public class WeatherResponse
    {
        [JsonProperty("product")]
        public string Product { get; set; }

        [JsonProperty("init")]
        public string Init { get; set; }

        [JsonProperty("dataseries")]
        public IEnumerable<WeatherData> Dataseries { get; set; }
    }

    public class WeatherData
    {
        [JsonProperty("date")]
        public int Date { get; set; }

        [JsonProperty("weather")]
        public string Weather { get; set; }

        [JsonProperty("temp2m")]
        public Temperature Temp2m { get; set; }
    }

    public class Temperature
    {
        [JsonProperty("max")]
        public int Max { get; set; }

        [JsonProperty("min")]
        public int Min { get; set; }
    }
}

We are also going to set up a configuration section in our appsettings.json so that the base address can be mocked later.

{
  "WeatherClient": {
    "BaseAddress": "https://www.7timer.info",
    "TimeoutSeconds": 30
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "AllowedHosts": "*"
}

So now we need to set up the refit HttpClient. We can do this inside ConfigureServices. To make things nicer I often put the code inside an extension method:

public static class ServiceCollectionExtensions
{
    public static IServiceCollection AddWeatherClient(this IServiceCollection services, IConfiguration configuration)
    {
        var settings = configuration.GetSection("WeatherClient").Get<WeatherSettings>();
        services.AddRefitClient<IWeatherClient>().ConfigureHttpClient((sp, client) =>
        {
            client.BaseAddress = new Uri(settings.BaseAddress);
            client.Timeout = TimeSpan.FromSeconds(settings.TimeoutSeconds);
            client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
        });
        return services;
    }

Then you just need to add AddWeatherClient(Configuration).

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddWeatherClient(Configuration);
}

Lastly, we are going to call our API from our controller.

Normally I would create a service for this but keeping it simple for brevity. Our controller now looks like this:

[ApiController]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
    private readonly ILogger<WeatherForecastController> _logger;
    private readonly IWeatherClient _weatherClient;
    public WeatherForecastController(ILogger<WeatherForecastController> logger, IWeatherClient weatherClient)
    {
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        _weatherClient = weatherClient ?? throw new ArgumentNullException(nameof(weatherClient));
    }
    [HttpGet]
    public async Task<IActionResult> GetAsync(CancellationToken ct)
    {
        var weatherResponse = await _weatherClient.GetWeatherAsync(51.5074m, 0.1278m, ct);
        if (!weatherResponse.IsSuccessStatusCode || weatherResponse?.Content?.Dataseries == null)
        {
            _logger.LogError("Unexpected status code from Weather API {StatusCode}", weatherResponse.StatusCode);
            return new StatusCodeResult(StatusCodes.Status500InternalServerError);
        }
        return Ok(weatherResponse.Content.Dataseries.Select(weather => new WeatherForecast
        {
             Date = DateTime.ParseExact(weather.Date.ToString(), "yyyyMMdd", CultureInfo.InvariantCulture),
             TemperatureC = new Temperature { Min = weather.Temp2m.Min, Max = weather.Temp2m.Max },
             Summary = weather.Weather
        }));
    }
}

Now when we run our API and call the get endpoint we will get back the results from the actual API.

[
  {
    "date": "2021-05-21T00:00:00",
    "temperatureC": { "min": 9, "max": 11 },
    "temperatureF": { "min": 48, "max": 51 },
    "summary": "lightrain"
  },
  {
    "date": "2021-05-22T00:00:00",
    "temperatureC": { "min": 7, "max": 10 },
    "temperatureF": { "min": 44, "max": 49 },
    "summary": "lightrain"
  },
  {
    "date": "2021-05-23T00:00:00",
    "temperatureC": { "min": 4, "max": 12 },
    "temperatureF": { "min": 39, "max": 53 },
    "summary": "lightrain"
  },
  {
    "date": "2021-05-24T00:00:00",
    "temperatureC": { "min": 7, "max": 12 },
    "temperatureF": { "min": 44, "max": 53 },
    "summary": "rain"
  },
  {
    "date": "2021-05-25T00:00:00",
    "temperatureC": { "min": 7, "max": 12 },
    "temperatureF": { "min": 44, "max": 53 },
    "summary": "rain"
  },
  {
    "date": "2021-05-26T00:00:00",
    "temperatureC": { "min": 6, "max": 15 },
    "temperatureF": { "min": 42, "max": 58 },
    "summary": "rain"
  },
  {
    "date": "2021-05-27T00:00:00",
    "temperatureC": { "min": 6, "max": 18 },
    "temperatureF": { "min": 42, "max": 64 },
    "summary": "clear"
  }
]

Because we are calling the actual API we can’t test what happens with a different response from the API. This is why we need to mock out this external dependency in our integration tests.

Creating our Integration Test Project

I prefer to have my projects laid out in the following format:

src/
├── Project/
│ ├── Project.csproj
│
test/
├── Project.Tests/
│ ├── Project.Tests.csproj
Project.sln

After moving the project into a separate src folder we can then use the dotnet templates to create our xunit test project inside our test project folder.

dotnet new xunit

Lastly back in the route folder we are going to create our solution file and add our projects to it.

dotnet new sln
dotnet sln add src/*
dotnet sln add test/*

Next we need to add a few packages to our test project:

<PackageReference Include="WireMock.Net" Version="1.4.15"/>
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="5.0.6"/>
<PackageReference Include="FluentAssertions" Version="5.10.3"/>

We also need to add a reference to our main project as well as an appsettings.test.json file and a Resources folder that we will add in a moment. Your test project should look something like this:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp5.0</TargetFramework>
    <RootNamespace>WireMock.Net.Test</RootNamespace>
    <IsPackable>false</IsPackable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0"/>
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="5.0.6"/>
    <PackageReference Include="xunit" Version="2.4.0"/>
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.0"/>
    <PackageReference Include="coverlet.collector" Version="1.2.0"/>
    <PackageReference Include="WireMock.Net" Version="1.4.15"/>
    <PackageReference Include="FluentAssertions" Version="5.10.3"/>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\src\WireMock.Net.Api\WireMock.Net.Api.csproj" />
  </ItemGroup>

  <ItemGroup>
    <None Update="appsettings.test.json">
        <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <Content Include="Resources\*.*">
        <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
  </ItemGroup>
</Project>

Now go and create an appsettings.test.json file and a Resources folder. The Resources folder is where we are going to put the mocked responses for our API.

In your appsettings.test.json file put the following:

{
  "WeatherClient": {
    "BaseAddress": "http://localhost:50000"
  }
}

We will set up our wiremock on this port in a bit.

Set up the WebApplicationFactory

To be able to replace the call to our actual API with our WireMock version we need our API to pick up our appsettings.test.json file so that we can replace the BaseAddress. To do this we are going to create a WebApplicationFactory and use it as a Class fixture in our tests.

using System.IO;

using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.Configuration;

using WireMock.Net.Api;

namespace WireMock.Net.Test.Infrastructure
{
    public class ApiWebFactory<TStartup> : WebApplicationFactory<TStartup> where TStartup : class
    {
        protected override IWebHostBuilder CreateWebHostBuilder() =>
            WebHost.CreateDefaultBuilder()
            .ConfigureAppConfiguration((context, config) =>
            {
                config
                    .SetBasePath(Directory.GetCurrentDirectory())
                    .AddJsonFile("appsettings.json", true, true)
                    .AddJsonFile("appsettings.test.json", false, false);
            })
            .UseContentRoot(Directory.GetCurrentDirectory())
            .UseKestrel()
            .UseStartup<Startup>();
    }
}

Next, we are going to create a base class for our integration tests to simplify making calls to our API and reading the responses.

Note: we need to add an XUnit Collection so that the tests are run sequentially, otherwise we are going to have issues with conflicting port numbers when running our test.

using System.Net;
using System.Net.Http;
using System.Threading.Tasks;

using Microsoft.AspNetCore.Mvc.Testing;

using Newtonsoft.Json;

using WireMock.Net.Api;

using Xunit;

namespace WireMock.Net.Test.Infrastructure
{
    [Collection("sequential")]
    public abstract class IntegrationBase : IClassFixture<ApiWebFactory<Startup>>
    {
        protected HttpClient HttpClient { get; }
        protected IntegrationBase(WebApplicationFactory<Startup> factory)
        {
            HttpClient = factory.CreateClient();
            factory.Server.AllowSynchronousIO = true;
        }

        protected HttpRequestMessage CreateGetRequest(string url)
        {
            return new HttpRequestMessage(HttpMethod.Get, url);
        }

        protected static async Task<T> ReadResponseAsync<T>(HttpResponseMessage response)
        {
            var result = await response.Content.ReadAsStringAsync();
            return JsonConvert.DeserializeObject<T>(result);
        }
    }
}

Then we need to create our mock API for the weather endpoint.

using System;
using System.IO;

using WireMock.RequestBuilders;
using WireMock.ResponseBuilders;
using WireMock.Server;

namespace WireMock.Net.Test.Infrastructure
{
    public class WeatherFixture : IDisposable
    {
        protected readonly WireMockServer _mockApi;

        public WeatherFixture()
        {
            _mockApi = WireMockServer.Start(50000);
        }

        public void Dispose()
        {
            _mockApi.Stop();
        }

        public void Reset()
        {
            _mockApi.Reset();
        }

        public IRequestBuilder SetupGetWeather(string responseBodyResource, int statusCode = 200)
        {
            var request = Request.Create()
                .UsingGet()
                .WithPath("/bin/civillight.php*");

            var responseBody = string.IsNullOrWhiteSpace(responseBodyResource) ? new byte[0] : File.ReadAllBytes(responseBodyResource);

            _mockApi.Given(request)
                .RespondWith(
                    Response.Create()
                    .WithStatusCode(statusCode)
                    .WithHeader("content-type", "application/json")
                    .WithBody(responseBody)
                );

            return request;
        }
    }
}

In this class, we are starting the WireMockServer on port 50000. This is so that we know that our mock API is going to be running on localhost:50000 when we run our tests.

The SetupGetWeather method is used to set up the expected request and response. In our case, anything that matches /bin/civillight.php at the start is going to be mocked.

Then have the response, which is retrieved from a file and returned in the body. Remember that Resources folder we created earlier? That is where we are going to put our JSON responses.

Create a file called success.json in the Resources folder and put the successful response from the weather API in there.

{
  "product": "civillight",
  "init": "2021052100",
  "dataseries": [
    {
      "date": 20210521,
      "weather": "lightrain",
      "temp2m": { "max": 11, "min": 9 },
      "wind10m_max": 4
    },
    {
      "date": 20210522,
      "weather": "lightrain",
      "temp2m": { "max": 10, "min": 7 },
      "wind10m_max": 3
    },
    {
      "date": 20210523,
      "weather": "lightrain",
      "temp2m": { "max": 12, "min": 4 },
      "wind10m_max": 4
    },
    {
      "date": 20210524,
      "weather": "rain",
      "temp2m": { "max": 12, "min": 7 },
      "wind10m_max": 3
    },
    {
      "date": 20210525,
      "weather": "rain",
      "temp2m": { "max": 12, "min": 7 },
      "wind10m_max": 3
    },
    {
      "date": 20210526,
      "weather": "rain",
      "temp2m": { "max": 15, "min": 6 },
      "wind10m_max": 3
    },
    {
      "date": 20210527,
      "weather": "clear",
      "temp2m": { "max": 18, "min": 6 },
      "wind10m_max": 3
    }
  ]
}

Lastly we create our Integration test and make sure we get back the expected response:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Threading.Tasks;

using FluentAssertions;

using WireMock.Net.Api;
using WireMock.Net.Test.Infrastructure;

using Xunit;

namespace WireMock.Net.Test
{
    public class IntegrationTest : IntegrationBase, IDisposable
    {
        private readonly WeatherFixture _weatherFixture;
        public IntegrationTest(ApiWebFactory<Startup> factory) : base(factory)
        {
            _weatherFixture = new WeatherFixture();
        }

        public void Dispose()
        {
            _weatherFixture.Reset();
            _weatherFixture.Dispose();
        }

        [Fact]
        public async Task Given_weather_api_successful_returns_weather()
        {
            // Arrange
            _weatherFixture.SetupGetWeather("Resources/success.json");

            // Act
            var request = CreateGetRequest("/weatherforecast");
            var result = await HttpClient.SendAsync(request);

            // Assert
            result.StatusCode.Should().Be(HttpStatusCode.OK);
            var response = await ReadResponseAsync<IEnumerable<WeatherForecast>>(result);
            response.Should().HaveCount(7);

            response.Should().Contain(x =>
                x.Date == DateTime.Parse("2021-05-25") &&
                x.Summary == "rain" &&
                x.TemperatureC.Max == 12 &&
                x.TemperatureC.Min == 7);
        }

        [Fact]
        public async Task Given_weather_api_unsuccessful_returns_500()
        {
            // Arrange
            _weatherFixture.SetupGetWeather(null, 503);

            // Act
            var request = CreateGetRequest("/weatherforecast");
            var result = await HttpClient.SendAsync(request);

            // Assert
            result.StatusCode.Should().Be(HttpStatusCode.InternalServerError);
        }
    }
}

Now run the tests using dotnet test and you should see that your API is calling the mocked endpoint.

info: System.Net.Http.HttpClient.Refit.Implementation.Generated+WireMockNetApiClientIWeatherClient, WireMock.Net.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.LogicalHandler[100]
      Start processing HTTP request GET http://localhost:50000/bin/civillight.php?tzshift=0&output=json&unit=metric&ac=0&lon=0.1278&lat=51.5074
info: System.Net.Http.HttpClient.Refit.Implementation.Generated+WireMockNetApiClientIWeatherClient, WireMock.Net.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.ClientHandler[100]
      Sending HTTP request GET http://localhost:50000/bin/civillight.php?tzshift=0&output=json&unit=metric&ac=0&lon=0.1278&lat=51.5074
info: System.Net.Http.HttpClient.Refit.Implementation.Generated+WireMockNetApiClientIWeatherClient, WireMock.Net.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.ClientHandler[101]
      Received HTTP response headers after 107.6376ms - 200
info: System.Net.Http.HttpClient.Refit.Implementation.Generated+WireMockNetApiClientIWeatherClient, WireMock.Net.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.LogicalHandler[101]
      End processing HTTP request after 117.0377ms - 200
info: System.Net.Http.HttpClient.Refit.Implementation.Generated+WireMockNetApiClientIWeatherClient, WireMock.Net.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.LogicalHandler[100]
      Start processing HTTP request GET http://localhost:50000/bin/civillight.php?tzshift=0&output=json&unit=metric&ac=0&lon=0.1278&lat=51.5074
info: System.Net.Http.HttpClient.Refit.Implementation.Generated+WireMockNetApiClientIWeatherClient, WireMock.Net.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.ClientHandler[100]
      Sending HTTP request GET http://localhost:50000/bin/civillight.php?tzshift=0&output=json&unit=metric&ac=0&lon=0.1278&lat=51.5074
info: System.Net.Http.HttpClient.Refit.Implementation.Generated+WireMockNetApiClientIWeatherClient, WireMock.Net.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.ClientHandler[101]
      Received HTTP response headers after 13.8969ms - 503
info: System.Net.Http.HttpClient.Refit.Implementation.Generated+WireMockNetApiClientIWeatherClient, WireMock.Net.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.LogicalHandler[101]
      End processing HTTP request after 14.0039ms - 503
fail: WireMock.Net.Api.Controllers.WeatherForecastController[0]
      Unexpected status code from Weather API ServiceUnavailable

Passed! - Failed: 0, Passed: 2, Skipped: 0, Total: 2, Duration: 832 ms

Final Comments

As I showed in my last post you can easily mock an API using WireMock in a docker container. However, if you want to be able to test different responses using the same endpoint then, you need to set up Wiremock.net directly in your integration tests.

You can find the complete code for this tutorial on my GitHub page.

If you have any questions let me know in the comments below.

Mocking API calls using WireMock

Alex Hyett — Fri, 14 May 2021 17:00:00 +0000

It is rare in software development that you are building something in complete isolation from everything else. Generally, you are going to be making calls to other systems or components.

If you are lucky you are building against an existing API that you can test against. If you are working with other teams to similar deadlines, you generally need to agree on a contract beforehand and build against it. In this case, you are going to need to mock the API to be able to test what you are building.

This is where Wiremock comes in. Other tools that can be used for mocking APIs but Wiremock is the one I have the most experience with.

Running Wiremock in a Docker container

If you need to run your component locally against a mocked source then running wiremock standalone from a docker container is the best way to do that. It works similarly to the way you mock AWS by running localstack. You can also use Wiremock to mock AWS services but I will save that for another post!

To save us some time we are going to use this popular (30 million pulls) docker image, rodolpheche/wiremock-docker.

If you want a one-liner to get this up and running you can use this:

docker run -it --rm -p 8080:8080 rodolpheche/wiremock

When wiremock starts you should see something like this on the console:

 /$$ /$$ /$$ /$$ /$$ /$$
| $$ /$ | $$|__/ | $$$ /$$$ | $$
| $$ /$$$| $$ /$$ /$$$$$$ /$$$$$$ | $$$$ /$$$$ /$$$$$$ /$$$$$$$| $$ /$$
| $$/$$ $$ $$| $$ /$$ __$$ /$$__ $$| $$ $$/$$ $$ /$$ __$$ /$$_____ /| $$ /$$/
| $$$$_ $$$$| $$| $$ \__/| $$$$$$$$| $$ $$$| $$| $$ \ $$| $$ | $$$$$$/
| $$$/ \ $$$| $$| $$ | $$ _____ /| $$\ $ | $$| $$ | $$| $$ | $$_ $$
| $$/ \ $$| $$| $$ | $$$$$$$| $$ \/ | $$| $$$$$$/| $$$$$$$| $$ \ $$
| __/ \__ /| __/|__ / \ _______/|__ / | __/ \______ / \ _______/|__ / \__/

port: 8080
enable-browser-proxying: false
disable-banner: false
no-request-journal: false
verbose: false

Got to love the ASCII art!

You should be able to get a list of mappings (which will be empty to start with) using http://localhost:8080/__admin/mappings which will give us this:

{
  "mappings": [],
  "meta": {
    "total": 0
  }
}

Adding mappings using the API

You can use the wiremock API to add mappings, this is particularly useful if you want to test out examples on the go.

GET Request

Using postman you can import the following curl statement:

curl --location --request POST 'http://localhost:8080/__admin/mappings/new' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "method": "GET",
        "url": "/api/test"
    },
    "response": {
        "status": 200,
        "body": "{ \"test\": \"data\" }",
        "headers": {
            "Content-Type": "application/json"
        }
    }
}'

Running this should give you a 201 created response.

Now when we call http://localhost:8080/api/test we should get back the following:

{
  "test": "data"
}

All you need to do is change the request and response and you can mock any API call.

POST Request

Mocking GET request by URL is the simplest option but if you are doing POST requests you are probably going to want to mock based on the request body.

For example, lets mock the POST endpoint that maps based on the body content:

curl --location --request POST 'http://localhost:8080/__admin/mappings/new' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "method": "POST",
        "url": "/api/test",
        "bodyPatterns" : [{
          "matchesJsonPath" : {
             "expression": "$",
             "equalToJson": "{ \"test\": \"banana\" }"
          }
        }]
    },
    "response": {
        "status": 201,
        "body": "{ \"test\": \"banana created\" }",
        "headers": {
            "Content-Type": "application/json"
        }
    }
}'

Just to prove it works we are now going to map another one based on a different body (fruit).

curl --location --request POST 'http://localhost:8080/__admin/mappings/new' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "method": "POST",
        "url": "/api/test",
        "bodyPatterns" : [{
          "matchesJsonPath" : {
             "expression": "$",
             "equalToJson": "{ \"test\": \"apple\" }"
          }
        }]
    },
    "response": {
        "status": 201,
        "body": "{ \"test\": \"apple created\" }",
        "headers": {
            "Content-Type": "application/json"
        }
    }
}'

You can test it out by running the following:

curl --location --request POST 'http://localhost:8080/api/test' \
--header 'Content-Type: application/json' \
--data-raw '{
    "test": "banana"
}'

curl --location --request POST 'http://localhost:8080/api/test' \
--header 'Content-Type: application/json' \
--data-raw '{
    "test": "apple"
}'

This is just a simple example, if you need something more advanced, check out the request matching documentation for more examples.

Resetting Mappings

If you want to reset all your mappings at any point then you can do so by sending a POST request to http://localhost:8080/_admin/reset

curl --location --request POST 'http://localhost:8080/__admin/reset'

Adding mappings using files

The other way to add your mock mappings is by using files that are loaded when the container is started. This is particularly useful if you want other developers to be able to run your code.

I prefer to run all my docker containers using docker-compose. That way all people need to do is run docker-compose up and everything will be set up for them.

Your docker-compose.yml file should look something like this:

version: '3.4'

services:
  wiremock:
    image: rodolpheche/wiremock
    container_name: wiremock
    volumes:
      - ./ __files:/home/wiremock/__ files
      - ./mappings:/home/wiremock/mappings
    ports:
      - 8080:8080

When you run docker-compose up it will create a __files and a mappings folder in your current directory that are mounted to the same folders in your container.

The mappings folder is where you save your json mappings. Each file is exactly the same as the body of the request that we were sending earlier.

For example:

{
  "request": {
    "url": "/api/test",
    "method": "GET"
  },
  "response": {
    "status": 200,
    "body": "{ \"test\": \"data\" }",
    "headers": {
      "Content-Type": "application/json"
    }
  }
}

If you want you can even just save the mappings we created by API earlier by sending a POST request to http://localhost:8080/__admin/mappings/save.

curl --location --request POST 'http://localhost:8080/__admin/mappings/save'

If you update any of the files that are saved in the mappings folder then you will need to run a reset before they take effect.

curl --location --request POST 'http://localhost:8080/__admin/reset'

The __files folder can be used to serve static files. For example if we put a simple index.html file in this folder like this:

<!DOCTYPE html>
<html>
  <body>
    <h1>Mocked File</h1>
    <p>Something interesting here</p>
  </body>
</html>

Then go to http://localhost:8080 you should see this:

Final Thoughts

Using mocks can be a great way to simulate the behaviour of external APIs. Wiremock is one of the simplest ways to get started. That being said, I have found that wiremock isn’t the fastest way to mock an API. I recently had to mock an FX rate endpoint with a set of static FX rates. It was much faster to do this with a simple API written in Node.js.

There is another version of Wiremock that I use regularly called Wiremock.Net which is particularly useful when writing .NET integration tests. I will cover that in my next post as it is a bit more involved than this one.

Using ngrok to test local websites and APIs

Alex Hyett — Wed, 07 Apr 2021 12:00:00 +0000

Often when I am creating a new website, I want to see how it is going to look on an actual device like my phone or tablet. You can use Chrome Web Tools for mimicing a device but it isn’t the same as an actual phone.

Chances are however that if you try and access the service using your IP Address on another device it will likely be blocked by your operating systems firewall. If you want to share the website or API with someone not on your network then you need to jump through lots of hoops such as forwarding the port on your router.

This is where ngrok comes in. It allows you to expose your local service or website over a secure tunnel.

Installing Ngrok

Ngrok couldn’t be simpler to install.

Head over to ngrok.com and sign up for an account (it’s free).
Download ngrok for your operating system. They support Windows, Mac (Intel and Arm), Linux (Intel and Arm) and FreeBSD.
Unzip the downloaded zip file
Connect ngrok to your account ./ngrok authtoken <auth token here>

Using Ngrok

Once Ngrok is installed it is simple to use.

First start up your application so that it is running on your local machine. If I run this Gatsby website locally it runs on port 8000.
Start a tunnel to your local port (8000 in this case) ./ngrok http 8000. You should see an output like this:

Ngrok gives you the http and https address that you can connect to and you will see the incoming HTTP requests on the console.

From their you can access the service running locally on your phone, tablet or even give it to someone else to try out.

You can also access a Dashboard locally on port 4040 if you want to get a detailed look of the requests and responses going to your service.

Ngrok is free for single use quick demos, if you need something more such as custom domains then you will need the paid account which starts at $5 per month.

Final Thoughts

Ngrok is one of those services that every developer should have in their toolbox. The dashboard is especially useful if you want to monitor the requests going to your service without another tool like fiddler. The free plan is good enough for most use cases, if you need more than the paid options are reasonable.

Using GitHub Actions to Deploy to S3

Alex Hyett — Fri, 26 Mar 2021 12:00:00 +0000

Recently I went through the process of setting up Drone CI on my Raspberry Pi. The plan was to use my Raspberry Pi as a build server for this website as well as other projects.

However, the Sharp image library that Gatsby uses to resize images doesn’t natively work on ARM processors. I did try to compile it in the docker image but I couldn’t get the correct version to be picked up when running the builds. On top of that the builds were incredibly slow on my already overworked Raspberry Pi 2B+.

So I started looking for x86 based single board computers (SBC) that would be able to run them. However, most of them aren’t that great and start at around £100, a bit more than a Raspberry Pi (the LattePanda looked quite good if you are interested).

At that price point, I might as well just run Drone CI on AWS in a micro EC2 (t2.micro £73 a year or free for first year).

Then I remembered, we have been using GitHub Actions at work and they are surprisingly good and cost effective. Really I am a bit late to the party with this!

Cost of GitHub Actions

If you have a public repository then you can run GitHub actions for free. Yes free!

Most of my code however is not public and for private repositories GitHub give you 2,000 minutes per month free (3,000 on a pro account). These builds also run with a generous 2 cores and 7 GB of RAM, a lot more than a t2.micro!

It takes around 2 - 3 minutes to build and deploy my blog to S3, so I am not likely to run out minutes each month with the frequency I post.

This is assuming your code runs on Linux which mine does. If you need to run on Windows then you only get 1,000 minutes a month. If for some reason you need your build to run on MacOS then that drops to 200 minutes.

Essentially at 2,000 minutes and $0.008 per minute on Linux, GitHub are giving us $16 dollars of runtime for free per month. As Windows and MacOS are more expensive to run you will use up your free credits quicker.

What can you do with GitHub Actions

With GitHub Actions you can pretty much do anything you would normally do on a build server.

Running builds and tests
Starting up docker containers
Run integration tests

On top of that it comes with tons of community made plugins as well.

Deploying a Gatsby website to S3

So this blog is hosted on Amazon S3 with Cloudfront. At the moment, this website costs me around £0.50 a month to run, which I can’t complain about (I have shared my terraform build in a previous post if you are interested).

However, until now I have been pushing up changes to my S3 bucket locally from my machine (yuck).

My current build process consists of a few commands that are set up in my package.json file.

yarn lint which runs ./node_modules/.bin/eslint --ext .js,.jsx --ignore-pattern public .
yarn build runs gatbsy build
yarn deploy runs the following command to upload to S3 and invalidate my Cloudfront cache.

gatsby-plugin-s3 deploy --yes; export AWS_PAGER=""; aws cloudfront create-invalidation --distribution-id E5FDMTLPHUTLTL --paths '/*';

The above is assuming I have also run yarn install at some point in the past and I have credentials set up on my machine to allow me to upload to Amazon S3.

GitHub Workflow File

To be able to use GitHub actions you need to create a workflow file under .github/workflows in your project.

This is what my workflow.yml file looks like:

name: Deploy Blog

on:
  push:
    branches:
      - main
jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: 12
      - name: Caching Gatsby
        id: gatsby-cache-build
        uses: actions/cache@v2
        with:
          path: |
            public
            .cache
            node_modules
          key: ${{ runner.os }}-gatsby-alexhyett-site-build-${{ github.run_id }}
          restore-keys: |
            ${{ runner.os }}-gatsby-alexhyett-site-build-
      - name: Install dependencies
        run: yarn install
      - name: Run Lint
        run: yarn run lint
      - name: Build
        run: yarn run build
      - name: Set AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: eu-west-1
      - name: Deploy to S3
        run: yarn run deploy

I am going to go through each of these steps so you know what they do.

This is my scripts section in my package.json file. I reference these in my workflow to keep things consistant.

"scripts": {
    "lint": "./node_modules/.bin/eslint --ext .js,.jsx --ignore-pattern public .",
    "develop": "gatsby develop",
    "build": "gatsby build",
    "clean": "gatsby clean",
    "serve": "gatsby serve",
    "deploy": "gatsby-plugin-s3 deploy --yes; export AWS_PAGER=\"\"; aws cloudfront create-invalidation --distribution-id E5FDMTLPHUTLTL --paths '/*';"
}

Define Build Environment and Timeout

build:
  runs-on: ubuntu-latest
  timeout-minutes: 10

Here we specify what operating system our workflow is going to run on. Here I am running on linux so I pick the ubuntu runner. You also have the option of Windows or MacOS:

windows-latest
macos-latest

You can find a list of runners in the GitHub Docs.

I have also specified a timeout of 10 minutes. It is good practise to set a timeout so that a malfunctioning workflow doesn’t eat up all your free minutes.

Define when to run this workflow

on:
  push:
    branches:
      - main

As it is only me contributing to this blog, I just have the workflow set to run when something is committed to the main branch. So if I merge a PR or commit straight to main it will kick off a build and deploy my blog.

Checkout Code

- uses: actions/checkout@v2

You will definitely need this step, as it checks out you code into the runner.

Setup Node

- uses: actions/setup-node@v2
  with:
    node-version: 12

As Gatsby relies on Node we need to set it up. I am using version 12 so I set that up here in the script.

Cache Folders

- name: Caching Gatsby
  id: gatsby-cache-build
  uses: actions/cache@v2
  with:
    path: |
      public
      .cache
      node_modules
    key: ${{ runner.os }}-gatsby-alexhyett-site-build-${{ github.run_id }}
    restore-keys: |
      ${{ runner.os }}-gatsby-alexhyett-site-build-

As you may know most React applications end up with a huge number of packages in the node_modules folder. If you are starting from a fresh project, a considerable amount of time is taken up downloading these packages. This will impact your build times and therefore your free minutes on GitHub.

For Gatsby we also want to cache the .cache folder and public folder so we don’t have to rebuild these each time.

GitHub action caches work by specifying a key which we use to find the cache. In this case, my key will look like this:Linux-gatsby-alexhyett-site-build-675994106. As I have specified restore-keys it will also match any keys starting with Linux-gatsby-alexhyett-site-build-.

If I chose to build on Windows instead then it would create a new cache key. There are various methods you can use to create a cache key which you can find more about on the repository.

Caches are kept while they are in use but if not used in 7 days they are removed.

Install Yarn Packages

- name: Install dependencies
  run: yarn install

This installs all of the npm packages require by my website. As the node_modules folder is cached this will be quicker after the first run.

Run Lint

- name: Run Lint
  run: yarn run lint

Check for any lint errors in my code before continuing. Lint is set up as script in my package.json file and is the equivalent to ./node_modules/.bin/eslint --ext .js,.jsx --ignore-pattern public .

Run Build

- name: Build
  run: yarn run build

Another script in my package.json equivalent to gatsby build.

Set to AWS Credentials

- name: Set AWS credentials
  uses: aws-actions/configure-aws-credentials@v1
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: eu-west-1

To be able to push to AWS and invalidate the Cloudfront cache we need to provide access keys and specify the region. Access keys are added using GitHub secrets in your project. I will cover how to set these up in a bit.

Upload to S3 and Invalidate Cache

- name: Deploy to S3
  run: yarn run deploy

Finally we need to push the built files in the public directory up to S3 and invalidate the Cloudfront cache. I do this with another script set up in package.json file.

Which is the equivalent to running this:

gatsby-plugin-s3 deploy --yes; export AWS_PAGER=""; aws cloudfront create-invalidation --distribution-id E5FDMTLPHUTLTL --paths '/*';

In case you are wondering, the export AWS_PAGER=""; command is so that the AWS CLI doesn’t prompt you to press enter after the invalidation has been done. This was a breaking change that the AWS team introduced recently.

Setting GitHub Secrets

The workflow script specifies a couple of secrets, ${{ secrets.AWS_ACCESS_KEY_ID }} and ${{ secrets.AWS_SECRET_ACCESS_KEY }}.

Ideally, you should create a user with just enough permissions to be able to perform what we need and no more.

To do this we need to create a new policy by logging on to AWS Console and going to IAM > Policies.

Then we need to create a policy with the following JSON, replacing yourdomain.com with the name of your S3 bucket for your website.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AccessToGetBucketLocation",
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::*"]
    },
    {
      "Sid": "AccessToWebsiteBuckets",
      "Effect": "Allow",
      "Action": [
        "s3:PutBucketWebsite",
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::yourdomain.com",
        "arn:aws:s3:::yourdomain.com/*"
      ]
    },
    {
      "Sid": "AccessToCloudfront",
      "Effect": "Allow",
      "Action": ["cloudfront:GetInvalidation", "cloudfront:CreateInvalidation"],
      "Resource": "*"
    }
  ]
}

Once the policy is created you need to create a new user under IAM > Users. Give it a name such as githubuser and select programmatic access:

Next select “Attach existing policies directly” and pick the policy we created in the last step.

On the last page you will be given your Access Key Id and Secret. You will only be shown this secret once so make sure you copy it down.

Lastly we need to add these secrets to GitHub. On your repository go to Settings > Secrets from their you need to add “New repository secret” and add in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Now the next time you commit changes to your main branch it will kick off the build:

There you have it. Much simpler than setting up your own CI tool either locally or in the cloud.

Conditional Compilation

This is a fairly simple example, although you can do a lot more with GitHub actions if you use docker or some of the action plugins.

If I wasn’t the only one contributing to this blog, I would also set up some of these steps to run on pull requests as well.

To do that I would set the on statement to run:

on: [push, pull_request]

And then add an if statement to the AWS and Deploy Steps:

if: { { github.ref == 'ref/head/main' } }

Final Thoughts

If you don’t have your own build server using GitHub Actions can be a great alternative. For public projects it is free and for private personal projects the 2,000 minutes is likely to be more than enough.

Getting Started with AWS Step Functions

Alex Hyett — Fri, 12 Mar 2021 12:40:00 +0000

I have recently been looking into AWS Step Functions. For those not familiar with them, Step Functions are Amazon’s way of providing a state machine as a service.

If you have used AWS you have probably used Lambda functions. Lambda functions allow you to easily put a bit of functionality into a stateless scalable component.

Of course, the issue with this is generally most business applications require some form of state to be useful. Business processes generally follow a workflow where it is only possible to complete one task once another has been completed.

AWS Step Functions allows you to easily set up scalable workflows for your serverless components.

Features of Step Functions

The main reason for choosing Step Functions rather than implementing your own is all the functionality you get out of the box. I have implemented workflows before and they can get complicated quickly.

Error Handling - AWS has error handling built-in and you can easily specify various catch blocks and tell what action you want to perform if something fails.
Retry Handling - Step Functions support retries as well to cope with the inevitable transient error.
Multiple ways to Trigger workflow step - there are a few ways you can kick off a workflow such as via API or a CloudWatch event. I will cover them in more detail later.
Workflow Progress Visualisation - The AWS Console will show you a visual representation of your workflow as well as showing the progress and whether a step has failed.

Main Benefits of Step Functions

Using AWS State Functions has many benefits:

Handles all error handling and retry logic - as mentioned in the features, all this work is already built-in.
Can handle complex workflows with hundreds of steps - Step Functions can cope with hundreds if not thousands of steps.
Copes with manual steps - many business processes require a manual review at some point.
Handles long-lived workflows up to 1 year - workflows using the standard type can last for as long as a year. So you don’t have to worry about your workflow timing out if there is a manual approval step.
Manage state between executions of stateless functions
Decouple the application work from business logic
Scales well and can perform parallel executions

Drawbacks & Limitations of Step Function

Of course, no technology is perfect and there are a few limitations you need to be aware of.

Decoupling all the workflow logic might make your application difficult to understand. Once you start linking together numerous individual Lambda components you need to makes sure that the workflow is well understood and documented.
Need to learn the AWS States Language which is based on JSON.
Locked into an AWS specific technology - Step Functions are unique to AWS and therefore if you plan on moving to a different cloud provider you will need to come up with a different solution.
Execution History limited to 90 days - if you need longer execution history you will need to rely on the CloudWatch logs.
Currently, only a few ways to trigger workflow transitions - not all the AWS services support triggering a workflow. The notable ones that are missing are from DynamoDB and Kinesis.
Data sent between workflow steps are not encrypted - if you want to pass sensitive data between steps you need to store it encrypted in either DynamoDB or S3 and provide a pointer in your step output.
Workflows can last up to 1 year on Standard and 5 mins on Express.
Execution History has a 25,000 hard limit - if you think the number of executions (including retries) is likely to go above this you will need to split up your workflow into multiple workflows. Luckily you can kick off other step functions from a step function.
Payload size limited to 32Kb - if you want to store above this amount you will need to use some other storage such as S3 or DynamoDB.
Only works natively with a handful of AWS services - if you need other applications to be triggered this will need to be done with a lambda wrapper.

AWS Step Function Workflow Types

When you are setting up a step function you will be given 2 options for the type of workflow you want, Standard or Express.

Standard

This is the default and is best if you have any long-running or manual steps in your workflow. This is best for low volume systems (requests measured per minute rather than per second).

Workflows can run for up to one year
2,000 executions per second
4,000 transitions per second per AWS account
Priced per state transition
Supports Activities & Callbacks - these are for long-running services that you don’t want in a lambda.

Express

If you have a high throughput system then you need the express workflow type. The express type is priced differently and does not support long-running processes.

Workflows can run for up to 5 minutes
100,000 executions per second
Nearly unlimited state transitions
Priced on the number of executions, duration and memory consumption
Doesn’t support Activities or Callbacks

AWS Step Function Pricing

The cost of using step functions will depend on whether you have chosen Standard or Express as your workflow type.

Standard Pricing

Standard workflows are priced at $0.025 per 1,000 state transitions. So a workflow with 10 states, executed 100 times will cost you $0.025.

If you have a high throughput system that executes your workflow 1 million times a day. That is 10 million state transitions so would cost $250 a day (or $91,250 a year!).

Express Pricing

Express workflows have 2 levels of pricing. They are charged at $1.00 per 1 million workflow executions ($0.000001 per request) and you are also charged for the workflow duration.

$0.00001667 per GB-Second ($0.0600 per GB-hour) for the first 1,000 hours GB-hours
$0.00000833 per GB-Second ($0.0300 per GB hour) for the next 4,000 hours GB-hours
$0.00000456 per GB-Second ($0.01642 per GB-hour) beyond that

Calculating pricing for express workflows is a bit harder but depending on your throughput it could save you a lot of money.

AWS give a few pricing examples on their website which might help you work out the pricing for your application.

Overview of Step Functions

Now you have seen some of the benefits of step functions, let’s have a look at how we set one up.

Triggering a Step Function

The whole point of having step functions is being able to automate a manual workflow. So naturally, you aren’t going to want to trigger these workflows manually via the AWS Console.

Luckily, AWS comes with several ways that you can trigger Step Functions:

API Gateway - using StartExecution from an API Gateway and pass in the payload from a POST call to your step function.
Cloudwatch/EventBridge Events - using EventBridge or Cloudwatch you can trigger step functions when another event occurs. Such as a file being uploaded to S3.
Amazon SDK - You can also trigger StartExecution using the SDK either from a Lambda function or another application.
Other Step Functions - it is good practice to split up large workflows into smaller ones that you can trigger using the task state.

Amazon States Language

To be able to write step functions you need to learn Amazon’s States Language. Don’t worry it is basically just JSON.

Here is a simple one state hello world example.

{
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Pass",
      "Result": "Hello World!",
      "End": true
    }
  }
}

The workflow starts with StartAt which indicates which step it is going to first.

You then have all of your states for your workflow inside the states property. Note, even though this has multiple items it isn’t an array.

You can indicate whether a state is an end state or passes on to another state using End and Next. For example, we could split this state up into a Hello state which then passes on to a World state.

{
  "StartAt": "Hello",
  "States": {
    "Hello": {
      "Type": "Pass",
      "Result": "Hello",
      "Next": "World"
    },
    "World": {
      "Type": "Pass",
      "Result": "World!",
      "End": true
    }
  }
}

This isn’t a great example as we aren’t doing anything with the output of the first state. The result of this step function will just be “World!“. I will get into how we can make this better using States.

Step Function States

As we saw from our Hello World example a state is made up of a Type, and either Next or End. The other parameters are dependent on the state Type.

The other fields you can use on any state are:

Comment - you can add comments so others know what your workflow is doing (and you in a few weeks)
InputPath & OutputPath - these allow you to select a portion of the input to pass on to your state or output to pass on to the next state. I will cover these in more detail in the Inputs and Outputs section below.

There are 8 different states we can use to make up our workflow.

Pass

The pass state we saw in our hello world example. As the name suggests this state doesn’t do any work but can be used to manipulate the data from the last state before passing it on to the next.

As we did with our example you can hard code results using the Result parameter. This can be an object as well as a string.

Such as:

"Result": {
    "x-datum": 0.381018,
    "y-datum": 622.2269926397355
 },

More useful is the Parameters option which allows you to construct an object using values from the previous state result.

For example, if you have the following output of your last state:

{
  "product": {
    "name": "T-Shirt",
    "details": {
      "color": "green",
      "size": "large",
      "material": "nylon"
    },
    "availability": "in stock",
    "cost": "$23"
  }
}

But your next state needed the information in this format:

{
  "product": "T-Shirt",
  "colour": "green",
  "size": "large",
  "availability": "in stock"
}

You can do this using the Parameters field as follows:

"MapValues": {
  "Type": "Pass",
  "Parameters": {
    "product.$": "$.product.name",
    "colour.$": "$.product.details.color",
    "size.$": "$.product.details.size",
    "availability.$": "$.product.availability"
  }
  "Next": "AnotherState"
},

You can use the Parameters field in the Task state as well to change the details you pass on.

Task

This is the state you are probably going to be using the most as it is the one that you need to interact with other AWS services.

You need to provide the Resource field which is usually the ARN of your AWS service.

"LambdaState": {
  "Type": "Task",
  "Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloWorld",
  "Next": "NextState"
}

At the time of writing AWS natively supports the following AWS services:

Lambda
AWS Batch
DynamoDB
Amazon ECS/AWS Fargate
Amazon SNS
Amazon SQS
Amazon Glue
SageMaker
Amazon EMR
CodeBuild
Athena
Amazon EKS
API Gateway
AWS Glue DataBrew
AWS Step Functions

You can find an up to date list of supported services on Amazon’s developer guide. If your service isn’t listed here than the easiest thing to do is to write a lambda function that calls it for you.

Choice

A workflow isn’t complete without some decisions being made. The choice state lets you specify conditions that need to be met to go to a particular state. For example, you might require manual approval if the value of the item is over a certain amount.

Given step functions are essentially infrastructure, you might not want to put too much logic into them. If you find yourself trying to write complex conditions it would be better to write this in a lambda function. At least that way you can write some unit tests around it (you do write unit tests don’t you?).

"ToBeOrNotToBe": {
  "Type": "Choice",
  "Choices": [
    {
      "Variable": "$.amount",
      "NumericGreaterThan": 1000
      "Next": "CEOApproval"
    },
    {
      "Variable": "$.amount",
      "NumericGreaterThan": 500
      "Next": "ManagerApproval"
    }
  ],
  "Default": "Approved"
},

Wait

Wait step does what you would expect. It tells the workflow to wait a certain number of seconds or until a specified timestamp.

"wait_ten_seconds": {
  "Type": "Wait",
  "Seconds": 10,
  "Next": "NextState"
}

"wait_until" : {
  "Type": "Wait",
  "Timestamp": "2021-03-14T01:59:00Z",
  "Next": "NextState"
}

To be honest, I can’t think of many reasons you might want to do this. If you are waiting for a process to finish you would be better off using a callback or an activity.

One example, could be an ordering system that allows customers to make multiple orders within a 30 minute period. If all the orders are placed within that time then they can all be dispatched as one order. In this case, after receiving an order you could have the system wait 30 minutes and then bundle other orders together as the next step.

Although you would have to stop subsequent orders from kicking off workflows of their own.

Succeed

If you want to complete your workflow early then you can use the succeed state. For example, you might have a choice state that determines nothing else needs to be done if a certain condition has been met.

"SuccessState": {
  "Type": "Succeed"
}

As this is a terminal state you do not need to specify a Next or End field.

Fail

Another terminal state you might want to use is when something unexpected happens. For example, a choice state could be used if any of the values coming through are null.

You can also optionally specify a Cause and Error message that will be returned. As with the succeed state, there is no Next or End field.

"FailState": {
  "Type": "Fail",
  "Cause": "Invalid response.",
  "Error": "ErrorA"
}

Parallel

Some steps in your workflow can be run in parallel. If they don’t depend on other steps and take in similar parameters you can run them at the same time with the parallel state. For example, you might want to do two API calls at once.

"RunTwoThingsInParallel": {
    "Type": "Parallel",
    "Next": "NextStateAfterParallel",
    "Branches": [
        {
          "StartAt": "APICall1",
          "States": {
            "APICall1": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:us-east-1:123456789012:function:GetInfo1",
              "End": true
            }
          }
        },
        {
          "StartAt": "APICall2",
          "States": {
            "APICall2": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:us-east-1:123456789012:function:GetInfo2",
              "End": true
            }
          }
        }
    ]
}

The states you want to run in parallel get added as branches. Each one has to be marked as an end state and you need to specify in the parallel state which state you want to run next.

Once all the states in your parallel state have finished executing the parallel state will return an array of results. The items in the array are in the order they appear in your parallel state. So in my example, the results of APICall1 will be the first item of the array and the results of APICall2 will be the second item.

Generally, you want to use a Pass state after a Parallel state to turn the array output of a parallel state back into a single JSON object.

Map

The map state is similar to Map in Javascript. It is used to iterate over an array and execute a state for each item in your array.

Taking the example from the AWS docs, if you have an input like this:

{
  "ship-date": "2016-03-14T01:59:00Z",
  "detail": {
    "delivery-partner": "UQS",
    "shipped": [
      { "prod": "R31", "dest-code": 9511, "quantity": 1344 },
      { "prod": "S39", "dest-code": 9511, "quantity": 40 },
      { "prod": "R31", "dest-code": 9833, "quantity": 12 },
      { "prod": "R40", "dest-code": 9860, "quantity": 887 },
      { "prod": "R40", "dest-code": 9511, "quantity": 1220 }
    ]
  }
}

We can use the following Map state to iterate over it:

"Validate-All": {
  "Type": "Map",
  "InputPath": "$.detail",
  "ItemsPath": "$.shipped",
  "MaxConcurrency": 0,
  "Iterator": {
    "StartAt": "Validate",
    "States": {
      "Validate": {
        "Type": "Task",
          "Resource": "arn:aws:lambda:us-east-1:123456789012:function:ship-val",
        "End": true
      }
    }
  },
  "ResultPath": "$.detail.shipped",
  "End": true
}

Here the InputPath is used to select just detail from the JSON input. Then we use ItemsPath to select the array we want to iterate over. You can then specify in Iterator the task you want to run for each item in your array.

In this case, MaxConcurrency is set to 0, which doesn’t place a quota on tasks being run in parallel. If you need the items in your array to be run in order then you can set MaxConcurrency to 1.

Inputs and Ouputs

There are several ways that you can manipulate the data that is sent as input and received as an output from your states. Not all of these fields are available in every state so you need to check the AWS docs before using them.

The only fields which are common to all states are InputPath and OutputPath.

InputPath

The InputPath is used to select the part of the state input that you want to pass on to your Task. For example given the following input:

{
  "result1": {
    "val1": 1,
    "val2": 2,
    "val3": 3
  },
  "result2": {
    "val1": "a",
    "val2": "b",
    "val3": "c"
  }
}

If you set the InputPath to $.result2 it would send the following as the state input.

{
  "val1": "a",
  "val2": "b",
  "val3": "c"
}

OutputPath

As we saw with InputPath the same can be done with the output. If you have a lambda that returns several fields but you are only interested in one of them then you can use the OutputPath to select the piece you are interested in.

Parameters

The Parameters field is the most useful one of all as it allows you to create new JSON inputs from your existing fields. I often use the Parameters field on the Pass state just to change the output of the previous state.

I gave one example of using Parameters when describing the Pass state. The other option is to use Parameters to turn the array output of the Map or Parallel state into a single object.

Take this output from a Parallel state that called two different APIs to get information.

[
  {
    "product": "T-Shirt",
    "color": "green",
    "size": "large"
  },
  {
    "product": "T-Shirt",
    "stock": 100
  }
]

You can use the parameters field to turn this into a single object.

"Parameters": {
  "product.$": "$[0].product",
  "color.$": "$[0].color",
  "size.$": "$[0].size",
  "stock.$": "$[1].stock"
}

ResultPath

You can use ResultPath to specify how the output of your state is going to appear. The default behaviour of the states is to just return the output. This means the original input to your state is lost.

This is the equivalent of having "ResultPath": "$" which means the output of your state is the whole result and replaces input.

This isn’t always useful. Instead, you could append the result of your lambda on to the input. Let’s say we want the output to be added to a result field. This can be doing using "ResultPath": "$.result".

Or if you want to completely ignore the output of your state and just return the input you can do "ResultPath": null.

ResultSelector

The ResultSelector is similar to the Parameters field but works on the output instead of the input. It allows you to create a new object from the output of your state. The ResultSelector is executed before the ResultPath.

For example, say we have a lambda that gets the stock for a particular product.

We have this input:

{
  "product": "T-Shirt",
  "color": "green",
  "size": "large"
}

And the output of our lambda function is:

{
  "product": "T-Shirt",
  "stock": 100
}

But we want to get an output that looks like this:

{
  "product": "T-Shirt",
  "color": "green",
  "size": "large",
  "availability": {
    "stock": 100
  }
}

To do that we need to have the following set:

"ResultSelector": {
    "stock.$": "$.stock"
},
"ResultPath": "$.availability"

Handling Errors in Step Functions

One of the main reason for using step functions is all the hard work of building a workflow engine is done for you. This also includes retries and error handling.

Retries

If a transient error occurs you generally want to retry a few times before raising an error. Adding in retries will make your application more resilient to failure.

"Retry": [ {
   "ErrorEquals": ["States.ALL"],
   "IntervalSeconds": 3,
   "MaxAttempts": 2,
   "BackoffRate": 2
} ]

If an error occurs the result of the above will be the following:

Error occurs
Waits 3 seconds
Retries the function. If it fails again then:
Wait 6 seconds
Retries the function. If it fails again then raise an error.

The ErrorEquals can be an exception raised from your application or one of the AWS error names which are: States.ALL, States.DataLimitExceeded, States.Runtime, States.Timeout, States.TaskFailed, or States.Permissions. You can read more about these in the AWS docs.

You can set MaxAttempts to 0 if you don’t want a particular error to be retried. For example, if your step function hasn’t got enough permissions to run then a retry probably isn’t going to help.

"Retry": [ {
   "ErrorEquals": ["States.Permissions"],
   "MaxAttempts": 0
}, {
   "ErrorEquals": ["States.ALL"]
} ]

If you only specify ErrorEquals and none of the other fields then it will retry 3 times with a 1 second interval and a backoff rate of 2.

Note: States.ALL has ALL in capitals, unlike the other error names. If you use States.All it won’t error it will just ignore it.

Catch blocks

Once the retries have been tried, then the execution will fail unless you set a catch block. The catch block lets you direct the workflow to an error handling state if a particular error is thrown. For example, you might want to send a message to say the workflow has failed.

"Catch": [ {
      "ErrorEquals": ["States.ALL"],
      "Next": "ErrorState"
 } ]

As with retries you can add multiple errors and have them go to different states if you want.

Activities & Callbacks

If you use the standard workflow you can use activities and callbacks. There is a subtle difference between activities and callbacks but each allows you to perform some work in your workflow asynchronously.

Activities

Activities were added first to step functions. Once you create an activity you can use the Task state to send work to your activity using the activity ARN.

The activity doesn’t do anything it acts as a queue of tasks that need to be picked up. To use activities you need to have a process that polls GetActivityTask. When an activity task is available you will get back a task token.

Once work has completed you can use the Amazon SDK or an API Gateway to call SendTaskSuccess or SendTaskFailure with your task token.

Callbacks

Callbacks work similarly but you don’t have to poll for a task token it is given to you when your task is called. Instead, you add .waitForTaskToken on the end of the ARN in your Task. The catch is that not all AWS services support callbacks. At the time of writing only the following AWS services support callbacks:

Lambda
Amazon ECS/Fargate
Amazon SNS
Amazon SQS
API Gateway
AWS Step Functions

As with Activities you then need to use Amazon SDK or an API Gateway to call SendTaskSuccess or SendTaskFailure with your task token once work has completed.

Callbacks and Activities can be used for manual approval processes or tasks that take a long time to run.

To implement a manual approval step you could do the following:

Get your step function to send a message to SQS with .waitForTaskToken
Set up an API Gateway that has approve and decline endpoints that send SendTaskSuccess or SendTaskFailure.
Write a lambda that picks up the SQS message and then sends an email via SES with links to your approve and decline endpoints with the taskToken added as a query string parameter.

Best Practices when using AWS Step Functions

AWS have a list of best practices that you should follow when writing step functions, I recommend you give those a read before trying out step functions in production.

In addition to Amazon’s own best practices, I have a few of my own after having some experience with step functions.

States should be idempotent

Chances are if a workflow fails you are going to want to replay it. Unfortunately, step functions don’t give you the ability to replay a workflow from a particular step.

There is a blog post on the AWS website that outlines how you can do it. However, the approach they outline involves dynamically creating a step function that has a choice state at the start that bypasses other steps. I don’t think that is a practical solution for a production environment.

You could add a choice state at the beginning of your workflow that has branches to every other state in your workflow. However, not only is this quite messy, you can’t isolate individual states in a parallel state.

Therefore the preferred approach is to ensure your states are idempotent and therefore can be run multiple times without any adverse effects.

Pass a unique id to your lambda functions

Trying to stitch together logs in a serverless system can be a nightmare if not done correctly. Especially if your lambdas are used outside of your step function. If a lambda raises an error how do you know what step function execution it relates too?

When you execute a step function it requires a unique execution id (unique within 90 days) to be sent through. You need to keep in mind that you may need to replay the workflow in which case you can’t use just the unique id from another system. You will need to append something else to it.

You can however pass this unique id down into your state input. The execution id is available in the context object.

Final Thoughts

Step functions are a great tool to have in your toolbox if you are working with a serverless architecture. I am certainly going to be using them on a few projects in the future.

I will set up a working example at some point in the future and write about it in another blog post.

Useful Docker Commands Worth Saving

Alex Hyett — Fri, 12 Feb 2021 11:30:00 +0000

I use docker every day. All the applications I write at work or at home end up in docker containers. Most of the time though, I am only running docker-compose up so when I need to do something more complicated I have to look it up.

So this post is a resource for me but I am hoping these commands will be useful for you too.

Cleaning up after Docker

If you have been using docker for a while you are going to end up with quite a few docker images hanging around. These commands should help clear them up.

Delete old containers

Most of the cleanup commands that are shared will clean out all of your containers. This command cleans up just those you haven’t used in a while:

docker ps --filter "status=exited" | grep 'weeks ago' | awk '{print $1}' | xargs docker rm

Delete untagged images

If an image isn’t tagged then you likely aren’t using it so we can get rid of these as well.

docker images | grep "<none>" | awk '{print $3}' | xargs docker rmi

Clean slate

If you really want to get rid of everything not currently running, including volumes and images you can use this one:

docker system prune -a --volumes

Building docker images

Most of you will already be familiar with docker build here are some of the less used docker commands.

Rebuild all docker-compose images with no cache

By default doing a docker-compose build will use cached images. If you want to make sure they are built from scratch and pull down images you need:

docker-compose build --no-cache

Rebuild just one service in your docker-compose

This one is especially useful on my Raspberry Pi where builds are a bit slower. If you know only one service has changed you can rebuild just that one.

docker-compose build --no-cache elasticsearch

Connecting to a docker container

If you need to debug why a container isn’t working you often need to connect to the running container.

Connect to a running container

To use this command you need to run docker ps to get a list of running containers, then use the container ID in the command below.

docker exec -it 0c7b2063b2a2 /bin/bash

If your container doesn’t have bash you might be able to use sh.

docker exec -it 0c7b2063b2a2 /bin/sh

Connecting to a stopped container

If you want to connect to a stopped container it is a little more complicated.

We first need to use docker ps -a to see the stopped container. Then you need to copy the container ID of the stopped container and commit it to a new image.

docker commit 0488e172aa70 test/image

You can then run it and connect to it using:

docker run -it --rm test/image /bin/bash

If the docker container has a faulty script at startup you may need to use this:

docker run -it --rm --entrypoint /bin/bash test/image

Copy files between container and host

Occasionally you might want to copy files to and from a running container. Using the Container ID again you can use the following commands.

Copy files FROM container

If you just want one file, for example the hosts file you can use:

docker cp 0c7b2063b2a2:/etc/hosts hosts

Or if you want a whole folder you can do this (where etc is a folder in the working directory on the host):

docker cp 0c7b2063b2a2:/etc/. etc

Copy files TO container

Similarly you can copy to the container to just by switching the parameters around:

docker cp fstab 0c7b2063b2a2:/etc/fstab

Final words

If I come across any other useful commands I will post them here. If you have any useful commands you use then please share them in the comments.