Using Skaffold for CI/CD with GitLab
Contributed by: @tvvignesh
Skaffold is a tool which is non-opinionated about the CI/CD tool you should use and acts as a platform agnostic solution for both development and deploying to production.
To facilitate building and deploying images in CI/CD, skaffold offers docker images which you can build on top of. If that doesn’t work for your use-case, you can make your own Dockerfile and pull in skaffold and its dependencies using curl.
Let us have a look at how we can use Skaffold with GitLab CI.
Step 1: Getting the project and Dockerfile ready
The first step is to obviously have your project which you want to deploy ready with a Dockerfile setup for the same.
Skaffold supports multiple builders like Docker, Kaniko, Bazel and more and if the builder you are looking for is not supported out of the box, you can script it out as well. We will use Docker in this demonstration.
For instance, if you would like to deploy nginx, your Dockerfile would look something like this:
FROM nginxinc/nginx-unprivileged:1.19.3-alpine
WORKDIR /app/server
COPY ./web ./web
EXPOSE 8080
CMD [ "/bin/bash", "-c", "nginx -g 'daemon off;'" ]
Skaffold can use your dockerfile to automatically build, tag and push images to the registry when needed.
Step 2: Choosing the deployment method
Skaffold supports deployment through kubectl, helm and a lot of other mechanisms. You can have a look at the complete list of deployers available here.
If you choose to deploy through kubectl, you must have all your yaml files ready, or if you would like to deploy through helm charts, you must have all your charts ready.
Step 3: Choosing the image and tag strategy
Skaffold automatically tags images based on different strategies as documented here: https://skaffold.dev/docs/taggers/. The tagging strategy used is configurable, so choose the mechanism which is right for you. By default, skaffold uses the git sha tagger.
If you are using helm as your deployer, you might want to use the helm image strategy if you would like to follow helm specific conventions, as skaffold will pass in all the details for you.
Step 4: Plugins (optional)
If you want to do some kind of processing before deployment like decrypting the secrets in the CI/CD pipeline, skaffold also supports plugins like Helm Secrets. So, if you would like to deploy secrets that have been encrypted using KMS or any other mechanism supported by SOPS you can actually set useHelmSecrets option to true and skaffold will handle everything automatically for you.
Step 5: The skaffold.yaml file
Your skaffold.yaml file will change depending on the way you would like to build and deploy your project. It is recommended that you set up skaffold and run it manually before you setup your CI pipeline since the CI pipeline can just call skaffold to do all the builds and deployments which can be tested out locally via skaffold.
A sample skaffold file can look something like this:
apiVersion: skaffold/v2beta8
kind: Config
profiles:
# Specify the profile name which you can run later using skaffold run -p <profilename>
- name: my-profile-name
build:
artifacts:
# Skaffold will use this as your image name and push it here after building
- image: asia.gcr.io/my-project/my-image
# We are using Docker as our builder here
docker:
# Pass the args we want to Docker during build
buildArgs:
NPM_REGISTRY: '{{.NPM_REGISTRY}}'
NPM_TOKEN: '{{.NPM_TOKEN}}'
deploy:
# Using Helm as the deployment strategy
helm:
# Pass the parameters according to https://skaffold.dev/docs/references/yaml/
releases:
- name: my-release
namespace: default
# Using Helm secrets plugin to process secrets before deploying
useHelmSecrets: true
# Location of the chart - here, we use a local chart
chartPath: ./charts/my-chart
# Path to the values file
valuesFiles:
- ./deployment/dev/values.yaml
- ./deployment/dev/secrets.yaml
skipBuildDependencies: true
flags:
upgrade:
- --install
Please have a look at the comments above to understand what the yaml does.
Rather, if you want to deploy via kubectl, your skaffold file can look something like this:
apiVersion: skaffold/v2beta8
kind: Config
profiles:
# Specify the profile name which you can run later using skaffold run -p <profilename>
- name: dev-svc
build:
artifacts:
# Skaffold will use this as your image name and push it here after building
- image: asia.gcr.io/my-project/my-image
# We are using Docker as our builder here
docker:
# Pass the args you want to Docker during build
buildArgs:
# Pass the args we want to Docker during build
NPM_REGISTRY: '{{.NPM_REGISTRY}}'
NPM_TOKEN: '{{.NPM_TOKEN}}'
deploy:
# In case we enable status check, this will be the timeout till which skaffold will wait
statusCheckDeadlineSeconds: 600
# Using kubectl as our deployer
kubectl:
# Location to our yaml files
# Refer https://skaffold.dev/docs/references/yaml/ for more options
manifests:
- k8/dev/*.yml
Step 6: Development & Testing locally
Before we move on to setting up the CI pipeline, we should test it out locally. Use skaffold run if you are deploying it to the cluster, skaffold build to just build and push the artifacts, skaffold dev if you are developing (this will enable auto reload on changes) or debug using skaffold debug
You can find docs about all these workflows here
Also, note that you can also set up file synchronization to enable faster development workflows. You can read more about that here.
Enabling file synchronization would avoid the need to rebuild the images repeatedly during development and testing and this can accelerate the inner dev loop.
Step 7: Setting up authentication with the registry
As you may already know, there are a lot of places where you can host images namely Docker Hub, GCR, GitLab Registry, Quay and so on.
The way you authenticate is specific to the registry provider of your choice. For eg. you may choose to do a username-password authentication with DockerHub or authenticate using service accounts in GCR and so on.
NOTE: Username-Password authentication is not recommended for CI/CD pipelines since they can be easily compromised and can sometimes be logged as well.
We will be using GCR in our example. You can refer this page to understand how to authenticate with it.
Step 8: Using DIND Service (Optional)
Docker images are typically not cached if you are running your builds/pipeline within Docker containers(i.e. DIND or Docker-in-Docker) and this can be very slow since your CI runner has to download the images again and again for every build even if some layers are already available.
NOTE: If you are not looking to run your pipeline within Docker containers or are fine with slower pipelines/pipelines without caching, you can completely skip this step and proceed to Step 9.
To avoid this, we can set up a DIND service in our cluster to enable caching and storage of image layers for us. A sample dind deployment file as deployed in Kubernetes can look something like this:
# Setup a DIND deployment within the Kubernetes cluster to cache images as we build images in our pipelines
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-dind
spec:
selector:
matchLabels:
app: my-dind
strategy:
type: Recreate
template:
metadata:
labels:
app: my-dind
spec:
containers:
- image: docker:dind
imagePullPolicy: Always
name: my-dind
ports:
- containerPort: 2375
name: my-dind
env:
- name: DOCKER_HOST
value: tcp://localhost:2375
- name: DOCKER_TLS_CERTDIR
value: ''
securityContext:
privileged: true
volumeMounts:
- name: my-dind-storage
mountPath: /var/lib/docker
resources:
limits:
memory: '2Gi'
cpu: '1000m'
requests:
memory: '1Gi'
cpu: '250m'
volumes:
- name: my-dind-storage
persistentVolumeClaim:
claimName: my-dind-pv-claim
Service File:
# Expose DIND as a service so that all the pipelines can access it
apiVersion: v1
kind: Service
metadata:
name: my-dind-svc
spec:
selector:
app: my-dind