DevOps.com

Where the world meets DevOps

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • DevOps Chats
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Communities
    • AWS Community Hub
    • CloudBees
    • IT as Code
    • Perforce DevSecOps
    • Rocket on DevOps.com
    • Traceable on DevOps.com
  • Related Sites
    • MediaOps
    • Container Journal
    • Security Boulevard
    • SweetCode.io
    • Digital Anarchist
    • DevOps TV on YouTube
    • DevOps Chat on SoundCloud
    • DevOps Connect
    • DevOps Dozen
  • Media Kit
  • Follow Us
    • Twitter
    • Google News
    • LinkedIn
    • Facebook
    • DevOps TV on YouTube
    • Email
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DevSecOps
  • Leadership Suite
  • Practices
  • ROELBOB
  • Low-Code/No-Code
  • IT as Code
  • More Topics
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps

Home » Blogs » Modernizing Continuous Integration

Harness CI/CD

Modernizing Continuous Integration

By: Ravi Lachhman on June 4, 2021 Leave a Comment

In the duality of continuous integration/continuous delivery (CI/CD), CI focuses on build automation. Leveraging CI practices for software delivery is normal for most organizations and can be seen as a solved problem. However, the demand for increasingly distributed applications has risen with the expansion of microservices, and development teams have the expectation that every commit should be a build.

Continuous integration has, surprisingly, become an unexpected bottleneck with these expectations. To begin modernizing CI, it’s important to first understand some of the challenges. Let’s dig in.

Challenges of Continuous Integration

Given that build and release candidates follow advancements in development technology closely — such as new languages, packaging and paradigms in testing the artifact — expanding the capabilities in continuous integration implementations can be challenging. With the introduction of containerized technology, the firepower and velocity required to build greatly increased.

Scaling Continuous Integration Platforms

The infrastructure required to run a distributed CI platform can be as complex as the applications they are building because of the heavy compute requirements. Consider how much a local machine’s resources are tied up during a local build and test cycle. Now, multiply that by the number of folks on a team or in an organization and you start to understand just how much compute power is being used. Distributed build runners are just one area that can be complex, and managing when new build nodes are spun up and spun down can depend on the platform/end user.

Keeping Up With the Technology Velocity

The adage of “the only constant in technology is change” is true. New languages, platforms and paradigms are to be expected as technology pushes forward. The ease of including new technologies in a heterogeneous build or accepting new testing paradigms, however, can be difficult for more rigid CI platforms. These legacy platforms were designed for only a small subset of technologies. With legacy or rigid approaches, the dependency management required to maintain technical velocity becomes a significant burden.

What it Takes to Modernize Continuous Integration

The simplest way to change continuous integration practices and platforms is to take what I call the four-pillar approach: platform infrastructure modernization, engineering efficiency strategy, test optimization and rapid pipeline development. Making strides in any of these pillars will put you on the path toward modernizing your continuous integration platforms and practices. Making strides in all of the pillars will set up your DevOps teams for significant success.

Platform Infrastructure Modernization

The location where the actual builds and packaging take place (e.g. build nodes/runners) do most of the heavy lifting and have a fairly elastic workload nature. Builds (e.g. a JAVA JAR build) and packaging (e.g. a Docker Image Compose) are compute-heavy tasks. Once the build and packaging are complete, the runners can sit idle. This shows the importance of having an ephemeral build node. The build node spins up during the task, then spins down or is destroyed after the build task is done, so as to not drain resources.

Engineering Efficiency Strategy

A core tenet of engineering efficiency is meeting your internal customers where they are. For software engineers, this means being as close to their tools and projects as possible. Like many modern pieces of application infrastructure, shifting left to the developer means being included in the project structure in source code management (SCM).

A common disconnect in CI platforms is dependency management. Over the past decade, for software engineers, this problem has been solved with package or build tools such as Maven, Gradle and NPM. Engineers need to simply define implicitly or explicitly what they need, and the dependencies will be resolved. Continuous integration tools can suffer from a disconnect, since we are leveraging several tools that potentially don’t have a common syntax. So, again, it’s important developers have close oversight on all the tools they need to complete their projects to reduce this friction.

Test Optimization

While initial confidence in getting tests into the CI pipeline expands, more tests — and sometimes inappropriate test coverage — is introduced by the ease of integration with the pipeline. An even harder problem to identify and rectify is flaky tests. A flaky test is one that both passes and fails periodically without any code changes. This becomes a twofold problem: increasing execution time and lack of confidence with flakiness. The most logical solution to avoid these issues is adding test optimization. A modern continuous integration solution should be able to visualize pipeline order, timings and overall execution to help identity (and eventually rectify) excessive coverage and flakiness.

Rapid Pipeline Development

Software is an exercise in iteration. The lower the barriers to entry for iteration to occur, the more gains in engineering efficiency and agility are achieved. Local builds happen dozens of times before reaching a committable stage, moving them forward to a dev-integration environment. It’s here where having a local environment is key. Oddly, most continuous integration pipelines are designed to run externally from a local machine. So, the ability to move to a local CI pipeline helps build confidence that the externally run pipeline will be successful.

Continue on Your Continuous Integration Journey

Continuous integration might seem like a solved problem for many organizations, but as with any technology, there is always room for improvement and modernization. Like any engineering efficiency paradigm, CI is a journey and the destination ever-evolving.

Recent Posts By Ravi Lachhman
  • DevOps and Networking: Working to Achieve Nirvana
More from Ravi Lachhman
Related Posts
    Related Categories
  • Blogs
  • DevOps Practice
    Related Topics
  • application modernization
  • CI/CD
  • continuous delivery
  • continuous integration
  • Harness
Show more
Show less

Filed Under: Blogs, DevOps Practice Tagged With: application modernization, CI/CD, continuous delivery, continuous integration, Harness

Sponsored Content
Featured eBook
DevOps: Mastering the Human Element

DevOps: Mastering the Human Element

While building constructive culture, engaging workers individually and helping staff avoid burnout have always been organizationally demanding, they are intensified by the continuous, always-on notion of DevOps.  When we think of work burnout, we often think of grueling workloads and deadline pressures. But it also has to do with mismatched ... Read More
« GitLab Acquires UnReview to Further AI Ambitions
Why AIOps Is Critical for Pandemic Business Recovery »

TechStrong TV – Live

Watch latest episodes and shows

Get The Top Stories of the Week

  • View DevOps.com Privacy Policy

Upcoming Webinars

OKRs: Friends or Foes?
Thursday, June 10, 2021 - 11:00 am EDT
How Instacart Uses Datadog and AWS CloudWatch for Real-time Monitoring
Friday, June 11, 2021 - 1:00 pm EDT
Who Benefits From Value Stream Management?
Tuesday, June 15, 2021 - 2:00 am EDT

Upcoming Webinars

Thu 10

OKRs: Friends or Foes?

Thu, June 10, 11:00 am - 12:00 pm EDT
Fri 11

How Instacart Uses Datadog and AWS CloudWatch for Real-time Monitoring

Fri, June 11, 1:00 pm - 2:00 pm EDT
Tue 15

Who Benefits From Value Stream Management?

Tue, June 15, 2:00 am - 3:00 am EDT

More Webinars

Past Webinars

Latest from DevOps.com

Get the Benefits of Low-Code Without the Risks
June 10, 2021 | Scott Bradley
Why API Quality is Top Priority for Developers
June 10, 2021 | Sheldon Warkentin
Observe Seeks to Put More IT Metrics in Context
June 9, 2021 | Mike Vizard
Leveraging AI in DevOps for Non-Linear Scaleup
June 9, 2021 | Deepak Gupta
Make a Plan for Test Automation
June 9, 2021 | Don Macvittie

Most Read on DevOps.com

GitLab Acquires UnReview to Further AI Ambitions
June 3, 2021 | Mike Vizard
Why AIOps Is Critical for Pandemic Business Recovery
June 4, 2021 | Sean McDermott
Modernizing Continuous Integration
June 4, 2021 | Ravi Lachhman
Security Heroes Wanted: Join Our Virtual DevSecOps Conferenc...
June 3, 2021 | Rebecca Auguste
Chicken Soup for the SLO
June 7, 2021 | Austin Parker

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

Download Free eBook

The 101 of Continuous Software Delivery
New call-to-action
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

© 2021 ·MediaOps Inc.All rights reserved.

Our website uses cookies. By continuing to browse the website you are agreeing to our use of cookies. For more information on how we use cookies and how you can disable them, please read our Privacy Policy.