GitOps Principle vs. Practice: The Cost of Accidental Deletions

This article explores the Argo CD mechanisms that help prevent accidental deletion of costly system resources.

A declarative system with a single source of truth is a core principle of any GitOps practice.

But we live in a world of cascading dependencies, inhabited by CustomResourceDefinitions, Kubernetes Operators, and IaaS abstractions such as CrossPlane.

In that world, Kubernetes resources represent far more than entries in an etcd database. They can be as complex as a PostgreSQL instance, as load-bearing as an S3 bucket storing terabytes of audit logs, or as foundational as a Kubernetes cluster.

Argo CD and Synchronization Policies

On top of that complex world, Argo CD introduces additional abstractions to represent the grouping of those resources:

• Applications are collections of Kubernetes resources deployed to a cluster.
• ApplicationSets represent collections of Applications.

When you combine all those layers, the accidental deletion of resources in a Git repository has the potential for triggering cascading deletions across the infrastructure, listed below in ascending order of impact:

1. Delete a custom resource representing an IaaS resource, such as an S3 bucket? There goes the S3 bucket and all of its data.
2. Delete a CustomResourceDefinition? There goes all Kubernetes resources matching the definition, along with all corresponding resources in the infrastructure.
3. Delete an Argo CD Application? There goes every managed resource in that Application. If one of the resources is a CustomResourceDefinition, you can do the math with the previous item.
4. Delete an Argo CD ApplicationSet? There goes every Application generated from the ApplicationSet.

The following sections cover the Argo CD mechanisms for preventing troubled Git merges from accidentally deprovisioning large swaths of your infrastructure.

These mechanisms range from requiring human confirmation to directives instructing Argo CD to ignore deletion requests.

It is important to note that vendors may also offer their own protection mechanisms, such as Kubernetes reclaim policies for persistent volumes and CrossPlane’s deletion policy for “Cluster” resources.

1. ApplicationSet syncPolicy: Preventing Application deletions

An ApplicationSet owns the Application resources it generates.

The default policy is governed by the ApplicationSet “syncPolicy” field. The default value for application synchronization (sync) adheres to the principle of a single source of truth:

1. If the ApplicationSet is deleted, all generated Applications are deleted.
2. If the generator no longer generates one of the applications it previously generated, that Application resource is deleted.

Something as simple as a typo in the target revision in a Git generator may result in an empty set of matching Applications. The default Argo CD configuration may cause the deletion of all Applications generated from that ApplicationSet, which in turn triggers a cascading deletion of all resources created downstream from that ApplicationSet.

If an ApplicationSet contains Applications representing expensive resources, that may be a costly mistake, so you may want to consider customizing the applicationsSync parameter of such ApplicationSets.

These are the options and respective explanations:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
...
spec:
  syncPolicy:
    # (default) Applications strictly reflect the output of the generator.
    # applicationsSync: sync

    # Prevents ApplicationSet controller from modifying or deleting Applications
    applicationsSync: create-only
    
    # Prevents ApplicationSet controller from deleting Applications. Update is allowed
    # applicationsSync: create-update

    # Prevents ApplicationSet controller from modifying Applications. Delete is allowed.
    # applicationsSync: create-delete

Note: The applicationsSync setting is only observed in Argo CD installations that are configured to accept policy overrides in the ApplicationSets. See this Argo CD document page for more details.

Reference: https://argo-cd.readthedocs.io/en/latest/operator-manual/applicationset/applicationset-specification/

2. ApplicationSet preserveResourcesOnDeletion: Preventing Application resource deletions

Overriding ApplicationSet applicationsSync parameters may lead to confusion about which Applications are still valid. After all, you may lose track of which Application resources are still being generated and which ones are orphaned by an applicationSync policy of create-update.

In such cases, you may prefer to have Argo CD delete the Application resources but use a non-cascading deletion policy for those Applications. In other words, you can instruct the applicationset controller to remove the Application resource from the cluster without applying the cascading deletion to all resources managed by that Application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
...
spec:
    # Prevent an Application's child resources from being deleted when the parent Application is deleted
    preserveResourcesOnDeletion: true

Reference: https://argo-cd.readthedocs.io/en/latest/operator-manual/applicationset/applicationset-specification/

3. Application non-cascading deletion

An Application resource may be created directly in a cluster, without being generated by an ApplicationSet.

There are valid reasons for wanting an Application resource removed from a cluster without triggering the deletion of resources contained in the Application.

In those circumstances, you can use the cascade=false parameter in the command-line interface or the Non-cascading option when deleting the application from the Argo CD user interface.

argocd app delete APPNAME --cascade=false

Reference: https://argo-cd.readthedocs.io/en/latest/user-guide/app_deletion/

4. Application syncPolicy prune: Preventing mass pruning of resources

Total deletion of an Application, covered in the previous sections, is only one of the ways an expensive resource may be deleted. There are cases where accidental deletion originates from the deletion of an individual resource in the Git repository.

In such cases, the resources deleted from the source Git repository may be removed from the system during the next synchronization operation.

By default, the pruning policy for resources is set to false, which marks an unusual deviation from the principle of a single source of truth. In other words, by default, Argo CD does not delete resources from a cluster when the resource is deleted from the source Git repository.

Regardless of its default value, this pruning policy can be overridden by setting it to true. I wanted to highlight this as a separate section in this article to establish the context for the following sections.

spec:
  syncPolicy:
    automated:
      ...
      prune: true       # Specifies if resources should be pruned during auto-syncing ( false by default ).
      ...

Reference: https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/#automatic-pruning

5. Resource Prune sync-option: Prevent individual pruning of resources

Assuming an Application pruning policy is set to true, there may be cases where you still want to prevent specific resources from being deleted from the system after they are removed from the source Git repository.

In those cases, you can use the Prune annotation in the resource, like this:

apiVersion: your_resource_group
kind: your_resource_type
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false|confirm

Note 1: You can also set this synchronization policy for an entire class of resources at the server level.
Note 2: The confirm option was introduced in Argo CD 2.14.

Reference: https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#no-prune-resources

6. Resource Delete sync-option: Prevent individual deletion of resources

The “Delete” option can add an extra layer of protection by preventing the resource deletion when the parent Application is deleted.

apiVersion: your_resource_group
kind: your_resource_type
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Delete=false|confirm

Note: The confirm option was introduced in Argo CD 2.14.

Reference: https://argo-cd.readthedocs.io/en/latest/user-guide/sync-options/#no-resource-deletion

AppProject syncWindows: Honorable Mention

Per Argo CD documentation for AppProject synchronization windows:

Sync windows are configurable windows of time where syncs will either be blocked or allowed. These are defined by a kind, which can be either allow or deny, a schedule in cron format and a duration along with one or more of either applications, namespaces and clusters.

With synchronization windows in place, system administrators can retain automated sync policies at the Application level, trusting that changes to the system due to Application changes will only happen at designated times and on designated systems.

In other words, this feature does not prevent accidental deletions, but may give system administrators additional time to examine diff changes and spot undesirable changes before the maintenance window starts.

Summary

A healthy GitOps practice requires some thinking about which resources can be managed according to a strict principle of a single source of truth.

The more resources you can manage that way, the less manual intervention you will need to clean up orphaned resources or manage drift between Git repos and the target environments.

If your GitOps practice is based on Argo CD, you can use this article as a guide to make informed decisions on where to implement the deviations and how to implement them.

The following table summarizes the different sections, their scope, and the pros and cons of adopting each approach:

Many services may become unused and forgotten in some corner of your Cloud provider bill.

Not Kubernetes.

Kubernetes hides its resource waste in plain sight, under a cloak of abstracted layers and compartments. From physical limits tracing their lineage to the early 1900s to a dizzying array of resource allocation techniques, it is virtually impossible to understand all nuances of how workloads are distributed across the environment.

In IT, there are three ways of dealing with problems in things we don’t understand:

1. Learn more about those things. Learning more is the theme of this article. The article does not cover everything, but it addresses the most common blind spots. Knowing what to avoid and why also helps one focus one’s observations about the system.
2. Reboot it. Whether restarting a pod, draining a node, or restarting a compute instance, “reboot it” is the Cargo Cult variation of resource allocation. In Kubernetes, this type of solution is often an illusion: a pod causing problems on a node will cause the same problems in its new hosting node.
3. Increase capacity. There are classes of problems that may require more capacity, but many other problems that just happen to be resolved with more capacity. For example, suppose a pod with an undeclared memory utilization of 32 GB keeps crashing the nodes on which the pod is scheduled. A system administrator unaware of the Kubernetes resource allocation model may decide to reallocate the entire node pool with an extra 32GB for each node.

While the “reboot” solution is simply a time-wasting distraction, the guesswork in resource allocation is an insidious shortcut, promoting thoughtless waste to a system requirement.

Kubernetes Waste is Structural

Virtualizing resources in production systems rely on workload owners specifying how many resources they may need.

Once again, Kubernetes is unique in that regard.

In Kubernetes, resource owners must specify the minimum resources they need (resource requests) and the maximum they can possibly use (resource limits.) And that is before we even start talking about affinity between pods and nodes.

Lost in that confusing mix, people tend to over-allocate resources to shield their workloads from resource starvation.

It is not unusual for a Kubernetes cluster to run with an allocated capacity several times larger than the workload usage within the cluster.

But how to avoid over-allocation?

• Should you set cluster capacity a few percent above the workload resource requests and hope that the capacity buffer is enough to handle eventual spikes?
• Should you set cluster capacity to meet the sum of resource limits for all workloads and watch the bulk of your IT budget being spent on pre-allocated capacity?

Indeed, we can auto-scale everything to try and allocate capacity on demand. At the top of the auto-scaling hierarchy, cluster auto-scaling takes care of running the minimum number of nodes, but there is still the question of individual node capacity.

The compute instances underneath those nodes come in pre-packaged sizes with fixed ratios between CPU and memory. Dynamically creating a new node with the wrong size is better than always running that same node, but it is still wasteful.

System administrators must find the ideal node size capable of running all workloads while avoiding surprises such as Cloud provider bills going up instead of down.

Avoiding those surprises requires a keen sense of where the typical blind spots lie, which brings us to the following sections.

Blind Spot #1. Limits Are Often Hidden, Rarely Abstracted

A running Kubernetes cluster sits atop a tall software and hardware stack, with each layer having its own capacity constraints and limits.

Limits lurking deep in that stack are a perennial source of seemingly mysterious messages indicating that pods cannot be scheduled on a lightly loaded node.

Those limits can be set at the operating system or the hardware level. As a software example, the operating system running a cluster node may have a maximum number of processes that can be concurrently open. For a hardware example, there may be a fixed number of PCIe slots in the computing device, which limits the number of storage volumes that can be mounted to a compute instance.

Few people will know all limits, but anyone managing Kubernetes clusters for a living should be familiar with these three:

1. Kubernetes limits. Kubernetes distributions have limits such as the maximum number of pods per node or the maximum number of nodes per cluster. Additionally, these limits are often different than those listed in the opensource documentation, such as those for ROSA Limits and Scalability
2. Compute Instance limits. Kubernetes nodes are compute instances with an operating system and various processes layered on top. A task as simple as requesting a persistent volume in Kubernetes may bounce into compute instance limits for how many storage volumes can be attached to the instance.
3. Infrastructure/account constraints. Cloud environments are virtually infinite in terms of capacity, but there are account-level limits that crop up when you least expect them. For example, you may not have problems creating the first few dozen clusters in your account. Then, the next request may fail because the cluster may need a new network gateway, and some cloud providers impose account limits on those¹.

Takeaway: Read through your cloud provider documents about the limits in these three abstraction layers (Kubernetes, compute instance, and IaaS account.)

Blind Spot #2: Burstable and BestEffort workloads

Planning is only as good as the quality of the data going into the planning.

You cannot decide on the best size for compute instances when you don’t know how much the workloads will need.

Containers inside a pod may have any combination of resource requests and limits, with different combinations of CPU and memory reservations defining the quality of service for the parent pod as follows:

• Every container in the pod has both resource request and limit set, and for each container, request and limit have the same value: Guaranteed.
• Resource request or limit not set in at least one of the containers in the pod: BestEffort.
• All other pods not in the two previous buckets: Burstable.

BestEffort and Burstable pods do not tell the cluster exactly how much CPU and memory they plan on using. In some cases, it may be a minor omission, such as not specifying CPU limits. Still, in other cases it may be a cluster-wrecking omission, such as not specifying memory requests.

Below is my scale of minor to significant ommissions in how workloads declare their resource needs, starting with the most severe:

1. No memory requests
2. No memory limits
3. No CPU requests
4. No ephemeral storage requests
5. No CPU limits
6. No ephemeral storage limits

There may be some debate at the bottom of the list, but not at the top.

In the worst case, workloads without any memory declaration (resources or limits) may crash several nodes in rapid succession, so you need to be at least aware of their existence.

There are many open-source and commercial tools that can analyze the historical resource usage of workloads in a cluster. These tools can identify workloads that may be underperforming due to a lack of resources and, more importantly, those that can destabilize other workloads.

For larger environments, some of these tools can continuously monitor resource utilization and create optimization plans for setting workload resource requests and limits. For smaller use cases, one can run a simple kubectl command like this or something more sophisticated, like Robusta's Kubernetes Resource Recommender.

Regardless of your chosen tool, you must understand how Kubernetes resource management works and scales across all dimensions. I wrote a comprehensive primer on Kubernetes scaling technologies.

Takeaway: Always identify all your Burstable and BestEffort workloads, then decide how much risk you are willing to take. There is no point in discussing budgets or ideal cluster node sizes when a significant share of workloads can simply start using cluster resources without notice or an upper limit.

Blind Spot #3: Overcome by Daemons

No matter the instance size, a fixed portion of the instance is reserved for the processes that make a node work within a larger Kubernetes cluster.

While those reservations can take up 1% or 2% of a node’s memory and CPU — somewhat more for storage — DaemonSets can take far more.

DaemonSets are typically used to run copies of pods on all (or some) nodes, augmenting the node capabilities with functions such as enforcement of security policies, log collection, observability, and storage.

A query to Prometheus, such as the one below, can show the memory resource requests for all DaemonSet pods across all nodes in a cluster — replace “memory” with “cpu” in the query to see memory requests:

sum by (node, daemonset) (
 kube_pod_container_resource_requests{resource="memory"}
 * on(pod) group_left(owner_name) kube_pod_owner{owner_kind="DaemonSet"}
)

These DaemonSets may get added gradually over time, and without asking the right questions about how many resources they will need, one can be blindsided by how little of each new node’s capacity remains usable by customer workloads.

Takeaway: Kubernetes DaemonSets can use considerable node capacity, especially when running on smaller compute instance sizes. Choose CPU and memory utilization thresholds for DaemonSets, then alert the operations team when that threshold is crossed.

Blind Spot #4: Bigger Instances, Bigger Root Volumes

While I argued in the previous section that smallish instance sizes can be wasteful, using larger instance sizes carries a different type of risk: storage limits.

As instance sizes increase, so does the number of pods per node.

Running more pods per node requires more disk activity, which brings us to the node root volume where that activity occurs. That root volume is where Kubernetes stores things like cached images and ephemeral volumes.

More pods mean higher demands in capacity, throughput, and transactions per second (IOPS). Of those three metrics, capacity is the easiest to exceed but also the easiest to observe — “disk full” messages have a way of making themselves known sooner rather than later. Throughput and IOPS tend to be the blind spots within the blind spot, with most shops not fully equipped to monitor or troubleshoot those problems.

So, if I double my instance size, should I double the capacity, throughput, and IOPS for root volumes?

In short, yes.

There is a word of caution here about whether your Kubernetes distribution exposes those settings to system administrators. For example, as of the writing of this article, ROSA allows the choice of disk size but does not allow the choice of throughput and IOPS reservations, so system administrators may have to locate the corresponding volumes in their AWS account to set those limits manually.

The last blindspot within the blindspot for this section is cost.

Cloud providers do not charge linearly for reservations. For example, AWS currently does not charge for volume reservations of up to 125 MB/s and 3000 IOPS when using general purpose volumes. However, if your larger compute instances require reservations above those free-of-charge limits, you will have new financial costs to consider.

Takeaway: Consider node root volumes’ three performance limits (capacity, throughput, and IOPS). Before choosing larger node sizes, you need to know whether you can increase those reservations and what their cost will be.

Blind Spot #5: Cheaper Instances, Fewer Volumes

Cloud providers do not optimize all compute instance sizes and families for containerized workloads. I often wonder if they even worry about instance sizes for Kubernetes.

As you choose larger instance sizes, the hardware limits in a typical server rack may become the limiting factor in the maximum size for those instances.

For example, as of the writing of this article, AWS EC2 has a limit of 28 volumes for its older instance generations regardless of instance size. As storage volumes are used to back persistent volumes, those 28 persistent volumes must be sufficient for the storage requirements from all pods scheduled in that node.

On the other hand, and still using AWS as an example, the more recent (and more expensive) instance series, such as 7 and 8, have volume limits that increase with instance size, making them more suitable for running Kubernetes workloads that require persistent volumes.

Takeaway: If your workloads require persistent volumes, read your cloud provider documentation and specs on storage limits. Then, assess whether your candidate instance types have enough “slots” for each node’s expected number of persistent volumes.

Blind Spot #6: Have You Talked to the Procurement People?

Technical folks (myself included) can be too quick to pull numbers from Cloud provider calculators and think they know what a compute instance costs.

For large companies, the real numbers are written in a contract negotiated for months between the company and the cloud provider. Those negotiated numbers are very different than those visible in external calculators.

The blind spot is that some of these discounted numbers may only apply to specific combinations of instance sizes and regions.

The rationale is that a cloud provider may be more willing to offer deeper discounts on a smallish 2x8 compute instance than a larger 64x256 instance. Those smaller instances can be more easily shoehorned in the unused crevices of a server blade on a rack — where they would go unused — while the larger instance may require an entire server blade.

Another related blind spot is that some discounted prices involve savings plans that require fixed reservation (or even prepayment) of capacity over a long period — months or years. Reducing wasted capacity reservation for your Kubernetes clusters may send account-wide utilization below that fixed reservation. Without a contract renegotiation on the minimum utilization, there may be no reductions in your Cloud provider bill.

Regardless of specifics about the discounted prices in that contract, initiating a contractual change before the contract ends is the equivalent of entertaining a new move right after unpacking the last box in your new house or apartment.

Takeaway: Before making capacity allocation decisions for your fleet of Kubernetes clusters, ensure you understand the actual cost of your options. You don’t want to find out your next Cloud bill went up after you spent considerable resources resizing all node pools in the fleet.

Summary

Reducing the total operation cost for a fleet of Kubernetes clusters is an optimization problem in a constrained space.

As with all math optimization problems, one must know their objective functions, decision variables, and constraints.

In plain English, one must know what happens to costs when the system settings change and which changes lead to unwanted results.

If individual nodes have the “wrong” resource allocation for the cluster workloads, even the best node auto-scaler algorithms and resource optimization tooling can only do so much.

Meeting workload objectives is the primary goal, but the total cost of ownership cuts directly into profits, so always consider instance pricing when making decisions. Depending on the workload, something like a dedicated pool of spot instances can be a sensible solution at a fraction of the cost.

And finally, aim for continuous improvements rather than ideal allocation. Simple changes can achieve considerable gains, while other optimizations may require more extensive performance analysis and reassessment of reservations at the workload level.

Footnotes

1. Sometimes, soft limits are set to avoid runaway resource allocation, and your cloud administrator can quickly increase them. Sometimes, hard limits require escalation or redesigning the entire system.

Let’s talk about revolution — the industrial kind.

I am not an economist, but I understand systems. When one thing changes, everything around it, from people to processes, changes to adapt.

We keep hearing about the impending moment when humanity achieves Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI), entering a period of prosperity and enlightenment devoid of poverty and disease.

However, humanity has been steadily progressing in economic growth, cultural enrichment, and advances in global health for millennia without AGI or ASI. Anticipating the continuation of that progress is not much of a prediction.

Several LLM advocates may argue this time is different because we have not seen anything like what is coming, begging the question of how much they have learned about technological progress in the past 60 years.

Spend a few hours learning about the last industrial revolution — the digital one in the 1960s — and it becomes clear that AGI and ASI would only create evolutionary automation plays in the long wake of those revolutionary waves.

I’ll explain.

Large Economic Models

Large Language Models (LLM) are AGI’s current favorite building block, and they excel in predicting the next word in a sentence.

They can be used in isolation or as the foundation for applications such as OpenAI’s ChatGPT or Anthropic’s Claude.

These enhanced LLMs can go beyond predicting words, having the ability to infer and follow steps from written instructions. On a side note, I think Apple’s recent paper on limitations of LLM reasoning failed to explore that aspect, though I still agree machine reasoning is far from being achieved.

LLMs are impressive feats of engineering in either isolated or combined forms, but industrial revolutions require dramatic changes in industrial processes, which in turn require new forms of thinking.

That is where LLM-based technologies fall short.

Artificial or natural, language processing is neither novel nor scarce. A new industrial revolution is not being held at bay by the lack of people or machines that can process text.

Machines generating emails or scientific slop are more likely to produce heat than value.

Before we can predict the revolutionary impact of AGI and ASI technologies in our economic systems, we must understand the structure beneath those systems.

While major economic theories may have been proven flawed in light of new developments — and I may be proven very wrong in the future — those theories were still directionally accurate.

For example, the labor theories of value prevalent in the 18th century anchored the value of goods to the number of hours spent producing those goods. Those theories were laid to waste during the 2nd industrial revolution, but engineering effort (which can be measured in hours) and energy utilization (a direct measure of work) still play a significant role in determining the value of goods.

On the opposite side of still-relevant theories, revolutions create new lines of sight to previously impossible economic concepts. A great example is the creation of the Internet, which made network effects possible and supported the creation of the platform economy.

Either through old or new theories, the problem for AI revolutionaries is not one of predicting the future but one of understanding the economics of why society may not always value technological advancement above all else.

If we want a perspective on the true meaning of the word “unprecedented” we need a brief history recap.

Liftoff — 1960-1979 — Where No Human Had Gone Before

In the 1960s, humanity witnessed the emergence of large-scale technological magic for the first time. Governments were pulling gigawatts of energy from rocks and corporations were modeling worlds using tiny bricks of processed sand.

Microchip-based computer systems guided us through the most important mission in history. Humanity was racing chariots with the gods and from 1969 to 1972, the Moon became Earth’s backyard.

By 1971, Russian landers and probes had touched Mars and Venus.

Nuclear technology threatened the world but was also on track to replace fossil fuels for industrial applications.

When combined, nuclear-powered probes and microchips opened the doors for deep space exploration.

In 1977, humanity started its journey toward the outer reaches of our solar system with concrete plans for making first contact with interstellar civilizations.

From the software side, we created programming languages and operating systems that power the core of the business world to this day.

In the ensuing decades, our popular culture pointed at flying cars and traveling to the stars.

From the infinitesimal to the infinite, there were no limits.

Between war and peace, our world would never be the same.

Ascent — 1980–1999 — The Internet Booster Burn

Deploying new reactors was expensive and time-consuming; by 1980, nuclear energy amounted to a mere 2% of global energy production. As a result, the world economy entered the 1980s reeling in the aftermath of the 1979 oil embargo.

But no matter, the 1980s also gave us the birth of the Internet (1983), IBM PCs (1981), Microsoft Windows (1985,) and even the first seeds of quantum computing technology.

The 1990s gave us the world-wide web and Linux (1993). Later in that decade, we also had the first commercial Bluetooth and WiFi devices.

Personal computers, wireless connectivity, a planet-wide network, and a (then) small company named Google brought it all together.

By the end of the decade, humankind took its second giant leap.

We were no longer connected virtually through a shared experience. We were connected, full stop.

We transitioned from making history to making stories. Moon landings and dreams of space colonization became distant memories. We became an old couple watching family photos, filling the void by looking inward.

Without communication floodgates between industry and consumers, the Internet fueled unrelenting economic growth, punctuated by Alan Greenspan’s “irrational exuberance” speech in 1996.

Our space dreams gave way to a pragmatism of space stations and low-orbit flights. We ramped up the Space Shuttle Program with its first crewed flight in 1981 and launched the International Space Station program in 1993.

On the energy front, the world grappled with the Chernobyl explosion (April of 1986), raising the specter of mass contamination of the entire European continent and reviving the public anxiety about the safety of nuclear power plants.

Protests led to moratoriums and immediate halts in orders for new reactors.

Installed capacity continued to grow but at a much slower pace, and decades of breathless growth in nuclear output came to an end.

The possibilities were still infinite, but we seemed content with rediscovering the world from our living rooms.

Reentry — 2000–2012 — Flight Turbulence and Hard Landing

Those years were marked by higher highs and lower lows.

The early 2000s were a nightmare on all fronts, from economy to technology.

In March 2000, the global markets peaked right before the popping of the dot-com bubble, triggering a seemingly endless free fall of all major indexes.

Those indexes would lose over 70% of their value by the end of 2002.

In between the highs and lows, we witnessed the literal collapse of the global seat of economic power on September 11th of 2001.

In the skies, the Columbia Space Shuttle disintegrated on atmospheric reentry (February of 2003,) reliving the traumatic events of space shuttle Challenger’s explosion during liftoff in 1986.

We could not catch a break.

On the energy front, gas fracking and shale oil extraction were breaking — and shaking — new ground. Excess production and lower prices put significant pressure on deploying new reactors.

Incidentally, the entire planet shook in March of 2011 with the Tohoku earthquake and the tsunamic aftermath that engulfed the coast of Japan.

With Fukushima reactors still melting under receding ocean waters, weary German officials made the landmark decision to phase out Germany’s nuclear reactors, joined by Switzerland in a similar fashion.

Nuclear power generation would never again break its all-time high of 2006.

Back to the economy, the markets remained unable to generate real value for several years and resorted to fake valuations, inflating a real estate bubble that culminated with the 2007–2008 near destruction of the global economy.

On July 21st, 2011, Space Shuttle Atlantis landed at NASA’s Kennedy Space Center in Florida, marking the end of the Space Shuttle Program. With SpaceX’s finances and first orbital flights on shaky ground for most of the 2000s, space exploration turned into space research, constrained to resupply missions to the ISS.

On the technology front, with the launch of Facebook (2004), YouTube (2005), and the delivery of the first iPhones (in 2007), we had the historical equivalent of the second invention of the press. While the physical presses created the large-scale ability to reproduce content for the masses, these technologies gave the masses the means to produce and distribute their own content.

The possibilities were still infinite, but we were more distracted than ever, and the cultural and industrial revolutions of the 1960s were finally over.

There were two extremely bright spots:

1. NASA solidified its shift from hovers to rovers, following Sojourner’s pathfinding mission to Mars with Spirit (January of 2004,) Opportunity (January of 2004,) and Curiosity (August of 2012.) India, EU, Russia, China, and UAE joined USA and the Soviet Union in successfully delivering orbiters and rovers to Mars orbit and soil.
2. Throughout the 2010s, several manufacturers released the first wave of commercially available Quantum computers. In time, these mind-bending machines are the only computing technology with the potential to rival the advent of microchips.

2012–2020 — IT’s Mission to the Clouds

I wrote this section as a separate chapter because we tech folks still had to make one final push in the long wake of the 1960s revolutionary winds.

This cycle is a stark lesson in the economics of supply and demand, acting as a cautionary tale about why we are overestimating the demand for LLM-based technologies.

With space colonization and fossil fuel replacements sputtering in the early 2010s, computer technology was still going, with Moore’s Law in full effect.

By 2011, software was eating the world, but the hardware was crowding us out.

Thanks to a combination of RFID, ever-cheaper WiFi routers, and a slew of IoT devices, computers were everywhere.

Those computers everywhere became a bit of a problem because we ran out of two critical resources:

1. Physical space. We were overrun by computer cabinets, racks, power supplies, and cables.
2. People. We didn’t have enough people to manage those servers.

Every small office and shop had a fleet of small towers spread from front to back. Unsightly metal boxes were laid down in improvised corners, tethered to an entanglement of cables connecting desks and rows of shelves.

In software development, we created parallel hierarchies between seniority and hardware ownership. At the top of that hierarchy, senior developers held the questionable distinction of managing a computing sprawl of workstations, mid-range servers under their desks, and often, high-powered test servers hosted in a secured area in the building.

We had IT people running around with carts and cables, performing various forms of inventory, maintenance, and repairs.

Cloud computing started to take root at the center of the technology industry because it solved the first problem (physical space) and helped with the second one (people). Economies of scale and an imperative of market capture created an entire industry of freemium service offerings backed by an unsustainable reliance on open-source quasi-products.

But even so, organizations began questioning the cost of running their operations and looking for ways to spend less on their cloud accounts, hardware, and personnel.

IT budgets went out of control, and jargon started shifting toward expressions like “business value,” “consolidation,” “time to market,” “mining existing data,” and many other terms indicating we had entered the “Town Planning” phase of the “Pioneer, Settler, Town Planning” cycle prescribed by Simon Wardley.

With limited growth caused by stiff competition and market cannibalization from hyperscalers, middleware providers accurately sensed the end of the decades-long market-capture cycle, prompting a series of defensive moves and monetization plays, from MongoDB in 2018 to Confluent’s Kafka in 2018 to Elastic in 2021. As more recent examples, we had HashiCorp’s Terraform in 2023, Redis Labs and, as a variation on the theme, WordPress’ fiasco with the WP Engine ban.

We held the keys to the virtual universe but the possibilities were no longer infinite.

Society, industry, and governments started questioning the price tag, much like they questioned the escalating prices of exploring the physical universe.

LLM-based technologies, however useful, are still just one of the many moving parts in the same overextended business case for the virtual universe.

The LLM Ballast

With these historic arcs in the rearview mirror, we can revisit the claims of an LLM revolution around the corner.

Even if we can get to scientific-research levels of AGI and ASI — a reality-defying if — we have not been short of ideas.

Before LLMs, and by several measures from patent applications to research papers, the generation of new ideas has already reached renaissance levels of creativity and innovation. LLM advocates may point at AI’s ability to generate more novel-sounding paper titles, which only reinforces the point that LLMs are very good with words.

From an economic perspective, replicating the exponential gains of the third industrial revolution requires technologies that significantly advance the productive processes in the real world, which in turn requires new forms of production and new energy sources.

The tech industry, however, has been trying too hard to extend its reach with a self-referential model in which the physical world is subordinate to the virtual one, first, with NFTs and now with LLMs.

And look, for all the doom and gloom of a nuclear-power industry that had seemingly lost the battle against renewable energy sources like solar and wind, the irony of a nuclear power renaissance driven by AI energy demand is not lost on me.

Setting aside the nuclear silver lining of our IT clouds, the economics of sending energy from the real world to build AI castles in the nether are terrible, backed only by the unsustainable math of market capture.

The Real World is the Real World

We may confuse our momentary fascination with electronic distractions for actual demand. However, the complex reality of our biological evolution remains a basic foundation of any economic and productive system.

Every systemic attempt at taking humanity deeper into virtual worlds, from 3D TVs to simulations of the physical world and immersive entertainment, has quickly plateaued at relatively disappointing numbers.

Case in point, the market for VR googles is about 20% of the market for earphones. And no, it is not the price points.

In “The Artificial Empathy of Generative AI,” I wrote about generative AI hitting the same dissonant chord with humans. In hindsight, I missed an essential point about the value of human interaction going beyond language and into the expectation of being the subject of attention from other human beings.

While I love LLM technology and have found potential usages for it, the economic foundations of an LLM revolution are shakily deposited atop improbable reversals in the dynamics of energy consumption and human evolution.

More than generated words, revolutions need enthusiasm and creativity.

That is why I think the next industrial revolution will be one of significant developments in energy generation and materials engineering. And because Earth’s atmosphere may be ill-equipped to dissipate energy generation at those scales, an eventual colony on the surface of Mars or the Moon may be the catalyst we need.

We may be decades or even centuries away from such a future.

In fact, I hope it does take long, just like computers took decades to shrink from office-sized machinery to miniaturized marvels.

A prolonged revolution means the technological imperatives of creativity and sustainability required to support those colonies would impact multiple generations of builders and aspiring engineers.

With the observed excitement over SpaceX’s recent successes with crewed flights and reusable boosters, one can only imagine the planet-wide euphoria of witnessing the first humans setting foot on Mars.

Whether on Mars or the Moon, a terraforming program would transform industry, society, and culture. Massive feats of engineering streaking the skies, from planes to rockets, have always inspired our collective attention like nothing else.

Looking up from our phones, we may finally wake up from our slumber and stop being distracted by machines that can imitate our past.

Intelligence?

We already have a surplus.

We have a GitOps repository that deploys the Grafana Helm chart and a few data sources introduced as secrets via sidecar datasources.

The structure of the secrets requires a string value key to contain the data source definition. That key name has no special significance as long as it is matched in the sidecar pod.

For example, here is a snippet exemplifying one of such secrets:

apiVersion: v1
kind: Secret
metadata:
  name: grafana-datasource
  labels:
    grafana_datasource: 'true'
stringData:
  # The name of this key can be anything
  object.yaml: |-
    apiVersion: 1
    datasources:
    - name: My datasource
  ...

Kubectl Apply versus Key Maps

We use Argo CD as our GitOps framework, which uses the equivalent of the “kubectl apply” command unless instructed otherwise.

You can override that default behavior, instructing Argo CD to replace resources instead of applying them.

In one particular update to the repository, after making multiple changes to the Secret, including the URL of the data source, we decided to replace the key name inside the data source secret — for reasons outside the context of this entry — so the new resource in the repository looked like this:

# Desired resource contents, pre-Argo CD sync
---
apiVersion: v1
kind: Secret
metadata:
  name: grafana-datasource
  labels:
    grafana_datasource: 'true'
stringData:
  # We changed the key name from "object.yaml" to "prometheus.yaml"
  prometheus.yaml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus datasource
  ...

We initiated the synchronization, and Argo CD completed it successfully, with no resources indicated as out-of-sync. In other words, the Secret had a nice green checkmark next to it.

However, Grafana still presented the old data source URL in the UI, which posed a bit of a mystery and triggered a round of troubleshooting and investigation.

Later, we learned that we still needed to restart the Grafana pods and clear out a couple of persistent volumes, but what intrigued us while troubleshooting the issue is that the live secret in the cluster now had the two keys, old and new:

# Live resource in the cluster, post Argo CD sync
---
apiVersion: v1
kind: Secret
metadata:
  name: grafana-datasource
  labels:
    grafana_datasource: 'true'
stringData:
  # Old secret data key, no longer in the Git repository
  object.yaml: |-
    apiVersion: 1
    datasources:
    - name: My datasource
  ...
  # New secret data key. The only one in the Git repository
  prometheus.yaml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus datasource
  ...

As mentioned earlier, Argo CD was perfectly happy to show the resource as synchronized.

The mystery ended once we went back to Kubernetes documentation to understand the Kubernetes algorithm for processing “kubectl apply” requests.

In essence, “kubectl apply” merges the input resource (the new Secret in this example) with the live resource (the Secret already created in the cluster), using different strategies for different types of fields.

- primitive: A field of type string, integer, or boolean. For example, image and replicas are primitive fields. Action: Replace.

- map, also called object: A field of type map or a complex type that contains subfields. For example, labels, annotations, spec and metadata are all maps. Action: Merge elements or subfields.

- list: A field containing a list of items that can be either primitive types or maps. For example, containers, ports, and args are lists. Action: Varies.

Since we changed a field name on a map, the expected behavior is “Merge” rather than “Replace.”

Mystery solved.

Different Content, No Drift?

I mentioned earlier that Argo CD synchronization was completed successfully and showed a green check “Synced” icon next to the resource.

It turns out that Argo CD did a great job considering the Kubernetes specification for “kubectl apply,” even though the literal content of the Secret in the live system was not the same as the content of the Secret in the git repository.

In other words, Argo CD only cared that the Secret data map had a key with the desired name (“prometheus.yaml”) and ignored the extraneous key (“object.yaml”).

Well done, Argo CD.

The Solution

Our use case called for that Secret in the cluster to be identical to the one in the Git repository, so we annotated the resource, instructing Argo CD to replace the resource (equivalent to invoking “kubectl replace”) instead of using the default “kubectl apply” behavior:

# New resource contents in the Git repository
---
apiVersion: v1
kind: Secret
metadata:
  annotations:
    # Instruct Argo CD to replace the live resource
    argocd.argoproj.io/sync-options: Replace=true
  name: grafana-datasource
  labels:
    grafana_datasource: 'true'
stringData:
  prometheus.yaml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus datasource
  ...

“I heard you. I still don’t believe you.”

Renaming keys on a Kubernetes resource is not the most common of operations, so a few people dealing with this situation wanted to run a small example using the kubectl CLI before they were ready to believe their eyes.

If you are also skeptical or if you need some backing to explain the issue to other people without having an entire Argo CD setup on hand, here is the snippet using only the CLI:

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Secret
metadata:
  name: apply-merge-example
  namespace: default
stringData:
  object.yaml: |
    apiVersion: 1
EOF

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Secret
metadata:
  name:  apply-merge-example
  namespace: default
stringData:
  prometheus.yaml: |
    apiVersion: 1
EOF

Now you can execute the following command to list the contents of the secret:

kubectl get Secret apply-merge-example -n default -o yaml

You will see the output with the two keys in the Secret:

apiVersion: v1
data:
  object.yaml: ...
  prometheus.yaml: ...
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: | ...
  name: apply-merge-example
  namespace: default
type: Opaque

Conclusion

Ultimately, the mystery stemmed from me letting my guard down with Argo CD’s great choice of default behavior for synchronization operations.

Using “Apply” strategies is the right approach for most situations. It allows you to co-manage resource contents with other sources — a necessity in the Kubernetes ecosystem — while also effectively producing results that are indistinguishable from the “replace” behavior — unless you are making certain types of changes to lists and maps (like the one we did).

Case in point, it was not the default “apply” behavior that caused our configuration problem. However, I can imagine different situations where an extraneous map key in a Secret or ConfigMap could cause problems, such as being processed by poorly written code that assumed the Secret had a single key or by code that used introspection of map keys to drive further actions.

I also acknowledge that the “replace” strategy poses problems in some cases. There may be cases where you don’t want Argo CD to come in and delete resource contents created by other sources. So, make sure you are not adding “replace” annotations to resources potentially co-managed by different sources (or, worse, manually).

Regardless of how you configure the synchronization strategy for Argo CD, either in the entire application or in individual resources, it pays to brush up on Kubernetes merge strategies and carefully consider when to use the synchronization method most adequate to the use case at hand.

“If you are nothing without this suit, then you shouldn’t have it.” — Tony Stark

With prompt engineering entering the professional vocabulary and the arrival of foundation models trained in programming languages, software engineers did not take long to look into prompt-assisted code generation.

As a software engineer with well-rounded experience in architecture, design, operations, and programming, I can see cause for enthusiasm for applying AI in software development — something I will cover in a follow-up story.

However, positive economics depend on how much effort I need to put into that explanation and the quality of the resulting answer.

Some problems are localized to small tasks; they are simple to describe and have trivial solutions. Their resolution is covered in training models to the point of overfitting.

Other issues involve complex relationships to different parts of the system, are hard to recognize, and their solutions include trade-offs that evoke explanations starting with “it depends.”

For those problems, anecdotal evidence reveals AI hallucination and failure rates that, produced by a human, would evoke the words “human resources,” “performance improvement plan,” or even “urine test.”

Since the story got impossibly long, I split it into two parts:

• Part 1 (this story): The economics of writing code. Here, I look at the short and long-term economics of solving non-trivial software engineering problems, using a real-world case study to compare the performance of a generative AI code assistant against the traditional, unassisted method.
• Part 2: The economics of creating and operating large systems. In this story, I will cover why we are missing out by focusing AI training efforts on coding assistance (where humans already excel) instead of the natural language surrounding the technical aspects of creating and operating an entire system.

Training for the Hard Problems

The principle of training AI on complex problems is still the same: show the AI enough examples to infer patterns and predict outcomes for situations similar to those where the data was collected.

When a programmer asks for a “Python function that reads the contents of a URL” to an AI model trained on 159 gigabytes of Python code source from 54 million GitHub repositories, we know a nearly identical solution is in that training set, right next to comments that are an almost perfect match to the wording of that request.

If that abundance of training material was not enough, the training data also contains several public-domain articles covering basic topics, in no small part because they are easier to write and consume.

But what happens when you work on a programming language that is not one of the most used programming languages in the world or use combinations of components glued together with bespoke, confidential source code written inside your organization?

In those situations, there is certainly less material to train the AI, introducing gaps in the training data and, consequently, reduced quality in prompt results — I will illustrate that point with a case study in the following section. Additionally, when dealing with copyrighted or access-controlled source code, some content may have to be left outside the training data, further increasing the gaps in domain coverage and decreasing the quality of the answers.

This is the point in the discussion where someone raises their hand and asks:

Have you tried tuning a foundation model?

Yes, fine-tuning a foundation model is a popular solution to adapting a trained AI model to the distinct characteristics of a specific training set. Fine-tuning a model, an activity where you start from a larger AI model and show it examples that are closer to the ones in your organization, costs a tiny fraction - think thousandths - of the cost of training a model from scratch.

Spending a tiny fraction of the cost while getting better results sounds like great economics, but keep in mind the starting point for those savings is in the realm of tens of millions of dollars and growing 4–5x every year.

That bargain price tag may still run in thousands of dollars per month in compute-costs alone.

“Non-sense. I can use a local LLM and train it on a couple of GPUs I bought on eBay and get the whole thing done for the price of a good dinner.”

The first part of the rebuttal is true — you can train or fine-tune a local LLM — but it is that second part that separates an inconsequential weekend project from actual software engineering.

The economics of fine-tuning a public foundation model are terrible.

Going from a prototype to a hosted solution can take as much as 27 times the effort spent on the prototype. Once that prototype is done, you have to factor in hosting costs, where you either rent compute power from a cloud provider or host your own hardware, then contend with the reality that fine-tuning is a means of improving the language portion of the model but not so much a way of adding knowledge to the final system.

After all, not even all examples in the world fully cover every possible function signature or CLI command, so you have to look at RAG patterns (Retrieval-Augmented Generation) to inject that knowledge into each prompt, which in turn means you will need a database, a query engine, and some time fine-tuning those augmentations to each prompt sent to the AI model.

And if you are serious about using that fine-tuned model augmented with a RAG system, you will need to develop a Machine Learning Pipeline around that solution. While designing that pipeline, you must decide on metrics to track the model performance, methods to collect those metrics, create processes for responding to feedback, and more processes for rolling out updates.

If that is not enough, you may also need to experiment with alternative fine-tuning methods to decide on the best approach for your effort.

The economics of muscling through that entire effort are those of a research project, with heavy upskilling on new technology and the added engineering effort of building and operating the final solution. And to cap the entire argument against the economics of training a localized model, there is always a chance the next version of that same starting foundation model will outperform your entire investment.

Assuming you are convinced that rolling out and operating your own model at the small scale of localized deployment is a lousy business, let’s look at a real example of using a code-generation assistant based on public foundation models — I used three different AIs throughout the exercise to ensure the results were consistent.

Case Study: Single Sign-On Configuration

This section is based on a recent real example where I had to package two components of a more extensive solution for an air-gapped environment (no direct connection to the Internet for security and regulatory purposes.)

I needed to configure an instance of HashiCorp Vault on a Kubernetes cluster to secure logins using an OIDC Client hosted on Keycloak. If you are not very technical, HashiCorp Vault is a service for managing secrets, and Keycloak is a service for managing access to other systems. In this solution, we want to use Keycloak to authorize who can access the secrets on Vault.

As a “hard problem,” this integration has a programming element combined with packaging and configuration aspects.

Securing access to Vault with an external access manager is not a rare system combination, but Vault can work with a dozen different authentication tools, including the public hosted offerings from the largest Cloud providers, so there is considerably less public content about using Keycloak than for those other alternatives.

When working on such a problem, I loosely follow the following steps, a framework I will later use to evaluate the AI’s performance.

1. Look for the documentation for both products.
2. Locate APIs and command-line interfaces.
3. Read through functions and parameters.
4. Map functions and parameters to the larger problem.
5. Draft an initial solution on a notepad (or blank page in a code editor.)
6. Iterate over the draft side-by-side with documentation.
7. Create a program to automate the instructions.
8. Write the documentation or notes for the component.
9. If I consider the solution generally usable, write a public blog or technote.

Phew! That seems like a lot of work, right? Time to ask an AI:

How do I configure Keycloak OIDC as an authentication method for HashiCorp Vault? Provide reference URLs after you give me the instructions.

The AI printed two pages worth of instructions, which were helpful and landed me in the middle of step 5 (draft an initial solution.)

Attempts to use the solution as drafted failed for several reasons, which is expected at that stage. However, unexpected issues forced me to backtrack to the earlier steps in the process.

The first problem was Vault complaining about not trusting the custom certificate in the URL for the Keycloak OIDC discovery endpoint.

I modified the initial prompt to indicate the Keycloak server used a custom certificate. Still, the AI seemingly ignored the alteration and did not produce any change in the responses. Assuming further clarifications would be equally ignored, I prompted the AI directly about how it would address the error message. The AI redeemed itself by showing an updated command with the correct parameter.

Next, while validating the solution and finally logging into Vault with a user stored in Keycloak, I noticed the Vault UI showing a numeric identifier for the user instead of the user’s email or full name. More than being friendlier to users, a recognizable name is also an extra layer of security in case the user has access to multiple accounts with different privileges (common in system operations.)

One could debate the importance of making that change, but I explained the problem in plain text to the AI, hoping for a solution. The AI generated another update for one of the commands, which did not produce the results I requested. At that point, I reverted to my usual step-by-step system and read through the command parameters.

The failed solution was because the parameter offered by the AI ("oidc_claim_mappings”) did not exist. The correct parameter name should be “claim_mappings”. Pause to note that while one would expect the Vault command-line interface to complain about the non-existent parameter, it did not.

At that stage, whatever time I had gained skipping the first four steps before drafting the initial solution was lost, and then some. As the joke goes: “One hour of debugging can save you five minutes of reading the documentation.”

I iterated over the draft following the documentation (“step 6”) and reverted to asking for the AI’s assistance with step 7 (“create a program to automate the instructions.”)

In that step, I asked the AI to generate a shell script automating the steps related to Keycloak, and it produced a respectable series of shell script commands based on low-level utilities such as “curl” and “jq.” On the less respectable side, the AI hardcoded all URLs and placeholders for credentials in the script, which would be terrible form for a human developer.

There was, however, a more severe and less visible problem: Keycloak already has an administrative CLI. Using direct access to REST endpoints with “curl” and processing JSON responses with “jq” worked but resulted in code that not only was harder to read (and maintain) but would also be sensitive to any future changes to the product APIs.

From an economics perspective, this module would have to be rewritten and retested when people discovered the glaring mistake, which would probably coincide with the first API change in Keycloak. There would also be reputational consequences (for me) in missing the existence of an entire administrative CLI while writing the code.

I played along with AI, not calling out the existence of that CLI. Curiously, the AI eventually switched its implementation strategy and offered a solution using the appropriate CLI without acknowledging it had been suggesting an incorrect approach in the preceding prompts.

With all the running bits completed, the AI’s assistance in step 8 (writing the documentation) impressed me the most.

I used the following prompt:

Write the step-by-step instructions for the following problem, grouping the information in sections titled “Overview,” “Prerequisites,” “Steps,” “Validation,” “Troubleshooting,” and “References.”

I want the result in markdown format that I can use on a README.md file.

On each step, add a “Command reference” link to the original documentation page describing the commands.

Here is the problem:

How do I configure Keycloak as an authentication method for HashiCorp Vault?

My Keycloak server is running on https://keycloak.server.com, using a custom certificate. The Realm name is “example”.

The Vault server is running on https://vault.server.com

I received an almost complete technical article in response. It had issues like a prerequisite section missing references to the product documentation pages and the same hallucinations seen in earlier steps. However, the AI assist addressed a common pain point for me and many other software developers: writing documentation at the end of a creative session, where the brain is wired to create solutions and not so much to explain the final solution — oh, and tired, too.

You can read the (lightly edited) version of the response. In full disclosure, I had to fix several bits of markdown for the URL links, but that only took a couple of minutes.

AI or Traditional Method. Who Won?

In this section, I break down each step in my usual process, calling out a “win” for the AI or “human method” in each step and then adding some commentary about each conclusion.

Step 1. Look up the documentation for both products

Win: AI

The AI gave me a usable link for each part of the process, embedded in the instructions, and with less friction than running a web search.

Web searches surfaced helpful blog entries and Stack Overflow links, but I had to vet through several entries (and ads), open the pages to skim through them, and then click a few links in the product websites to find the document sections I needed.

It is worth noting the traditional process has advantages in the long run — I’ll come to them later — but I am looking strictly at time spent and immediate results.

Step 2. Locate APIs and command-line interfaces

Win: AI

Similar to the previous step, asking for links in the context of the actual problem produced good results, with the advantage of having them embedded in the instructions close to the steps where the APIs and CLIs were being used.

I had to spend extra time writing down the problem for the AI than I would typically spend writing my search criteria, but that was not extra time relative to the entire activity.

Step 3. Read through functions and parameters

Win: Human Method

Reading manuals and understanding the technologies involved in the problem is a fundamental aspect of producing a workable solution with the lowest cost of ownership in the long run.

AI drafts may be acceptable on a disposable prototype but inexcusable on an actual product. The AI cannot create a semantic network of the product documentation in our brains or its own circuits, as evidenced by the hallucination of the “oidc_claims_mapping” parameter or the temporary blindspot surrounding the Keycloak administrative CLI.

Step 4. Map functions and parameters to the problem

Win: Human method

The AI drafts surfaced some parameters in the drafted solution, but those parameters were only a fraction of the whole. Reading through the documentation gave me a much better perspective on assessing the quality of the original AI draft and other alternatives and considerations for adapting the solution to a production system.

Understanding the overall range of configurations in the underlying technologies is the only way to judge whether the solution will be stable and have the lowest cost of operations in a production system.

Step 5. Draft a solution on a notepad (or blank code editor page)

Win: AI

After all iterations, the initial draft contained 90% of the final solution, an outstanding effort from a single-sentence prompt.

Using the traditional method of looking up examples through web searches would surface all the building blocks required to create such draft. Still, the AI could string them together much faster and generate what was basically a technical article with steps to solve the problem.

Step 6. Iterate over the draft side-by-side with documentation

Win: Human Method

I took over from the AI and did the iterations, partially because it could not improve upon its (admittedly solid) first draft but mainly because I lost trust in the AI’s accuracy after troubleshooting the hallucinated parameter for the Vault CLI and the scare with the temporary blindspot with the Keycloak CLI.

This experience matches what I heard and read from other senior engineers about how AI chats tend to plateau and even derail once you start to add more details about your request.

Step 7. Create a program to automate the instructions

Win: Human Method

Setting aside the initial AI blunder of ignoring the Keycloak administrative CLI, I liked how the AI was able to create a shell scripting boilerplate using the “getopt” command.

However, writing over a copy of an existing shell script in our project would have been more efficient due to their fit to our coding standards (such as copyright statements, naming conventions, message formatting, error handling, and others.)

Step 8. Write down the final instructions in a markdown file

Win: AI

The original write-up in step “5” felt like the skeleton of a technical article.

It is worth noting the AI had difficulty honoring my request to write the output in markdown format, switching back mid-text to its rich-text rendering format, no matter how specific I tried to be about the entire output using the markdown format.

Also, the AI “plateaued” in improving the quality of answers against refined prompts, such as ignoring my different wordings of a request to embed links to the product documentation in the prerequisite section.

Those issues were, however, minor relative to the overall content.

As long as one understands they should not spend too much time trying to keep refining prompts in an attempt to get the final version of the instructions, the AI saves time relative to starting from a blank page.

Step 9. Write a public blog or technote.

Did not attempt

I think the solution may be of general interest — for an admittedly niche audience — but I simply did not have the time. If I did have the time, I would still prefer to write it myself with the editorial assistance of an AI rather than defer the writing to the AI entirely.

With AIs able to write technical articles on the fly, I cannot imagine anyone willing to read through the result of someone else’s prompt rather than prompting an AI to write something bespoke to their particular problem.

The Short-Term (potential) Gains Are Appealing

In the short term, with four wins for each side and some loose counting of the overall time spent in the activity, I declared a tie on the economics of enlisting a generative coding AI to my workflow.

Different case studies with different technologies may swing the balance towards net gains or losses, depending on how well the training data covers the domain.

My exercise used a somewhat well-established product — Keycloak is a CNCF incubation project — and the de-facto standard for secrets management, so I am assuming good, if not extensive, coverage in the training data.

The AI initially missed a severe blindspot and hallucinated a few parameters. However, it made up for its mistakes in the natural language areas of identifying useful documents and writing documentation drafts.

Outside this assessment, the AI also routinely excels in the initial drafts for the short Python-based functions I tend to use in side tasks, such as analyzing the contents of our Git repositories, typically requiring only a few editorial changes.

I think focusing the tool’s usage on ancillary, simple tasks is the sweet spot for AI assistants. However, my enthusiasm for it is more of the incremental type — such as when I first mastered a data pivot in Excel — than the kind where I would consider quitting my job and starting a software company staffed with inexpensive robot developers.

The Long-Term Losses Can Be Devastating

In the long term, the entire debate about the gains and losses of prompt-based programming ignores a crucial byproduct of the problem-solving cycle: gaining knowledge.

There is a formative aspect in solving problems using traditional methods of investigation, discovery, abstraction, programming solutions, and validating the resulting code.

While investigating the building blocks for a solution, we are creating an entire domain model of those concepts, full of boundaries and trade-offs between things that work well under different circumstances.

Receiving an answer to a question will never make our brains tread the same steps of processing and assembling the building blocks of that answer. In that sense, quicker results committed to a code repository come at the expense of not building skills and maturing talent at the same pace.

That is a frightening trade-off backed by ambitious assumptions that AI capabilities grow faster than we erode ours.

For how long will use cases ending in ”make sure the results [of a prompt] are correct” still be valid in a system where asking for those results becomes the norm and we collectively atrophy the skills required to arrive at those same results?

Some technology enthusiasts make the case for generative AIs being a great learning tool. Still, I often hear those arguments from senior people learning yet another skill — backed by their substantial knowledge of multiple domains — rather than from those starting their careers.

Other rebuttals resort to a question turned argument: “Would you rather code in Assembly, then?”. Can people not see the distinction between adopting a higher abstraction language layer versus a cognitive process favoring interrogation over abstraction and reasoning?

There is also little acknowledgment that asking for help, a skill at the core of prompt-based coding, is an evolved skill in itself.

The general messaging may be of industrial revolution, but industrial revolutions unlock untapped resources, which is not the case here.

Generative AI, at its core, taps into existing resources already explored and consumed in different ways, relying on contentious mining of public-domain resources at the source and on use-cases cementing existing expertise at the point of delivery.

Along the way, the whole process endangers the early stages of the talent pipeline and threatens deskilling the very pool of content authors generating the raw material the AI depends on.

There has to be a better way.

The inspiration for this story started with another one, by David Caudill, titled Maybe Getting Rid of Your QA Team was Bad, Actually.

In that story, David takes the reader through a worrisome progression where organizations lose sight of quality over time and, in the process, progressively devalue the importance of QA engineering roles.

QA or not, there is a common theme of people observing fundamental changes in corporate priorities and wondering if their organization still has a place for them.

To use a traveling metaphor, unexpected changes in direction and surfaces can be disorienting and even hazardous.

The Map of Change

Tricky roads require maps, and when you need maps for anything that is not an actual road, look no further than Simon Wardley’s maps.

If you have never heard of “Wardley maps,” it is the best-kept secret this side of a business management school and a reliable way of understanding the landscape of any topic, whether you are looking at an emerging market or trying to figure out where the next career opportunities are heading.

In one particular map, loosely titled “Pioneers, Settlers and Town Planners”, Simon describes the flow of evolution and change within organizations.

Understand that model, and you will start recognizing orderly flows where you would typically see random events and decisions.

In that model, there are three stereotypes for how organizations, small teams, and individuals fit into the change flows:

• Pioneers deal with rare opportunities in ill-defined areas, with a high risk of failure and high future value in case of success.
• Settlers deal with growing markets, high profitability, and the need for adaptability to changing conditions in those markets.
• Town Planners deal with well-defined, already established areas. They focus on delivering high volume with discipline in process and cost controls.

For the remainder of this story, I will refer to Simon’s model after the stereotypes’ initials: PST.

Ugly Ducklings and Egos

Like in the story of the ugly duckling, even the most competent individual will underperform in the wrong setting.

While every large organization has a set distribution of resources across the three pillars of the PST model, individuals can only be excellent in one pillar at a time.

For example, someone tasked with writing a prototype for a new market should not be asked to observe rigid standards of DevSecOps discipline.

Conversely, a QA engineer who is excellent at creating automated processes and structured guidance for a growing product — aligned with a “settler” role — may struggle to contribute value in a research organization with shorter project spans and higher failure rates.

A few readers will try to make the case that an individual can be simultaneously excellent in two adjacent pillars, such as “Settler” and “Town Planner.” Before making that argument, however, they should make sure their example was not someone moving Kanban cards on a board while people worked on their email backlog — been there.

Unlike in the Danish fairy tale, a poor fit between individual tendencies and PST pillars runs deeper than the length of a feather or the shape of a beak.

How deep?

Take it from Jörg Heitkötter, who while remarking on an article about Simon Wardley maps, pushed right across the lines of Nature over Nurture, and contrasted personal interests against the dimensions of the PST model:

Ego vs. Company interest

It occurs to me that one interesting “feature” of pioneers, settlers, and town planners has not been stated or investigated in the model yet:

Pioneers tend to have an ego they put over the company; at the other end of the spectrum, Town planners tend to put the company over their ego and do what’s best for the company, while Settlers tend to be in equilibrium with regard to ego vs. company interests.

This might be helpful for HR people or managers to identify the right members for their teams, sorting out the fake Pioneers/Settlers/Town planners from the real ones.

This topic of personal and corporate interest opened my eyes to better ways of identifying and relating to a handful of team dynamics I would consider troubling otherwise.

What opened my eyes was the realization that shifting from one pillar to another may entail a change in personal values and in how individuals relate to their institutions. When people say they are not sure they can adapt to a different PST pillar, they may be balking at changing their entire outlook in life.

The Fractal of Change

The pattern of PST roles of change in an organization repeats itself in smaller parts, down to the individuals in that organization, forming a fractal dimension. In other words, a smaller part of an organization primarily focused on one pillar, such as research, still needs to apportion its resources in a PST framework.

Naturally, each pillar still needs to focus and specialize in its primary function, but within each pillar, smaller change flows require attention, too.

Balancing that allocation across all levels is the recipe for sustained success.

Neglect the exploration of the new and innovation stalls. Neglect the elaboration of a promising activity and growth stalls. Ignore the day-to-day work management, and long-term success becomes impossible.

That goes for individuals, too.

Balance it right, and the pioneering side of an enterprise may reveal new paths that eventually transform the entire business.

Look no further than my favorite example of sustained business transformation: Nintendo.

Nintendo has a tumultuous history spanning over 100 years, starting as a playing card company in the late 19th century.

Nintendo was successful for decades, then struggled, then partnered with Disney in the 1950s, found success again, and went back to struggling in the 1960s.

At some point during that struggle, Nintendo tried everything in sight to get out of the slump, from manufacturing food to vacuum cleaners and toys.

And how did Nintendo transition out of stagnation in the 1960s?

Through the creativity of a maintenance engineer turned product designer:

In 1966, Yamauchi [Nintendo’s President at the time], upon visiting one of the company’s […] factories, noticed an extending arm-shaped toy, which had been made by one of its maintenance engineers, Gunpei Yokoi, for fun. Yamauchi ordered Yokoi to develop it as a proper product for the Christmas rush. Released as the Ultra Hand, it became one of Nintendo’s earliest toy blockbusters, selling over hundreds of thousands units. […] Yokoi was soon moved from maintenance duty to product development.

Due to his electrical engineering background, it soon became apparent that Yokoi was quite adept at developing electronic toys. These devices had a much higher novelty value than traditional toys, allowing Nintendo to charge a higher price margin for each product. Yokoi went on to develop many other toys, including the Ten Billion Barrel puzzle, a baseball throwing machine called the Ultra Machine, and a Love Tester.

Fun fact: Nintendo still makes playing cards.

When Change Requires Theft

Earlier in this story, you may have noticed the “stealing” references between roles in the PST model, with settlers stealing from pioneers, town planners stealing from settlers, and pioneers borrowing components from town planning resources.

It may not sound good, but it makes perfect sense.

As pioneers find something with growth potential, it is time for settlers to make that growth happen. That transition is never smooth because settlers may appropriate the tools and techniques and even deprive pioneers of the credit. Look no further than the contentious debate on Elon Musk being a pioneer or a settler in the founding of Tesla Motors.

As a business reaches a certain status and becomes a commercial success, it is natural to “industrialize” the money-making sectors of that business, multiplying the profits per unit across more units. The search for differentiation and adaptation, a vital aspect of the settler routine, is the enemy of efficiencies and maximum gains.

And when everything is organized for maximum efficiency to the detriment of all else, it is time for the organization to carve out resources to form an expeditionary force of pioneers.

The circle of change is complete, and once again, the same goes for individuals:

• “That idea was mine” precedes the hard truth that the pioneer behind it was content to have the idea and not explore it.
• “They stopped investing in our project” precedes the reality that the project peaked a couple of years ago.
• “Some people get to have fun exploring new things” negates the reality that new ideas and opportunities keep the organization adaptable to changes in the environment.

Once again, the same goes for individuals, who may overstay in a productive role where their skills are a perfect fit for their project needs. In that stagnant arrangement, everything they know is useful to the project, their importance grows, and their careers advance.

As projects and products inevitably reach a plateau and start to decline, successful software developers may suddenly find themselves at the top of the wrong hill.

They miss their exit turn.

Master of Journeys, not Destinations

To bring the theory full circle toward practice, let’s use the dispirited trek of our QA colleagues in the story opening. Organizations focusing heavily on town planning activities need quality control more than quality assurance.

Conversely, portions of the organization pioneering new technologies may not have much output to worry about quality controls or assurances. Awareness of the territory tells QA engineers to seek roles in growth areas where quality matters and the nature of what is measured is constantly changing.

Yes, I am saying QA engineers, at their core, are settlers.

But can a settler-at-heart thrive in town planning? Yes, it is a fractal of change, remember?

As the business looks to streamline processes, often through automation, there is still room for _fewer_ individuals to develop new techniques to optimize the processes.

A Case Study in Adaptation

As a concrete example, let’s say you work in the QA activities for a mature product line. You then automate a process that simulates chaos in the environment running the product code. The simulation would include things such as turning off network firewall rules, restarting VMs, killing processes, and such.

Since these simulators already exist, building a new process around them is a “settler” principle (reusing existing technology developed by pioneers). And the new process can be applied to an organization in “town planning” mode, introducing a low-cost approach that can be embedded into other quality control pipelines.

Take that idea a step further; you can pair up with an SRE team to embed that new process into the early stages of a SaaS pipeline. Now, we have the potential to notify software developers about the high-availability characteristics of their code changes in the context of their pull requests.

Take that potential even further, and such technology can create chaos in a production environment and gauge its resiliency in the face of actual outages.

Is that pioneering?

Not really; Netflix did it some nine years ago. Remember, settlers steal from pioneers — and I am not talking about stealing credit, just the output.

There is Always a Path

If you are not a software developer but feel like software developers get all the credit, the way forward is to inject more coding into your work.

For example, if you are a technical writer and your product has an API, create a better way of validating code samples in documentation examples. If the product only has a graphical user interface, find a way to use LLMs to generate documentation pages while a bot monitors users iterating with the product interface. There is (almost) always a way.

And if you are an SRE, we are living through a golden age of opportunity for AI-assisted operations, with LLMs increasingly showing potential in helping people formulate commands during critsits or perform basic troubleshooting assessments.

Keep Moving, Keep Building

On a smaller scale, because the map of change flows is a fractal of smaller PST models, even being in the “wrong” PST pillar for your working style does not mean a life sentence of devaluation and disgruntlement:

• Even the most pioneering teams need some order.
• Even the fastest-growing businesses need standardization and stability to avoid chaos.
• Even the most profitable mature business can use innovation in smaller uncharted processes.

While these smaller actions can make you more valuable to your organization — and happier, too — the moral of the story is that when the only constant is change, becoming a specialist in navigating change is a core competence to succeed in our profession.

And now you have a map, too.

As long as you are willing to keep moving, your present situation is just a stop on the way to where the organization needs you.

If commerce downtown is too crowded or life as a shopkeeper is not for you, there is always money to be made transporting goods across town limits or erecting new buildings.

If something doesn’t exist, create it.

If it exists and is valuable, build upon it.

If something works, keep doing it, but keep looking for the next thing.

“How hard can it be to find a new Java developer?”

I was asked this question by our engineering director, frustrated with my lack of progress in finding a new software developer for our project.

By then, I had interviewed several candidates and concluded they would not be successful in the role.

Our team found a good fit a couple of weeks after the nudge, so I never had to explain the real reason why I did not recommend the previous candidates.

It was us, not them.

My reservation, which I helped the candidates understand, is that while they could meet the immediate requirements of the project — we would be fortunate to have them aboard — working with our technology stack was not an optimal choice for them in the long run.

With enough time passed and that entire project no longer active, I can explain my assessment using a structured framework.

In this story, I explain that framework and share a couple of personal anecdotes illustrating why it is relevant whether you are starting or well underway in your professional career.

The Framework: Hierarchy of Career Priorities

Let’s start with an adaptation of Maslow’s Hierarchy of Needs, which I’ll call the “Hierarchy of Career Priorities.”

Like in Maslow’s theory of layers of priorities before reaching a state of self-actualization — where one has the continuous desire to become the most they can be — my framework for career success works in the same way, with analogous layers as follows:

1. Employability. This foundational layer matches Maslow’s “Physiological Needs” layer. It represents the basic requirements for entering the workforce, including relevant education, training, degrees, and skills demanded by the job market.
2. Compensation and Stability. This layer represents having a steady job, adequate compensation matching the required time and skills to do the job, and a supportive work environment.
3. Growth and Development. This layer involves continuous learning, skill enhancement, and professional development. In this layer, a job allows and requires professionals to develop their skills to perform the role and retain their future employability. The layer also includes the viability and growth trends of the project’s business case.
4. Work-Life Balance and Well-Being. This layer is about balancing work and personal life, supporting the employee’s overall well-being. This layer focuses on maintaining a healthy work-life balance, managing stress, and prioritizing mental and physical health.
5. Purpose and Meaning. The pinnacle of the “Hierarchy of Career Priorities,” where individuals seek purpose and meaning in their chosen careers. This layer involves aligning personal values with the professional career, making a positive impact, and finding fulfillment beyond material and external rewards.

Ideally, employment choices should favor a good alignment of these layers. It is impossible to align them perfectly at all times, but awareness of eventual imbalances allows one to correct them sooner rather than later.

Here are a few common examples of such imbalances:

• A generous compensation package incentivizing people to sacrifice their well-being with long work weeks and stressful air travel routines.
• Overstaying in a role with a stagnant “growth and development” layer.
• Becoming overly skilled in an aging technology stack, trading growth in the current job for employability in the broader job marketplace.

I’ll reference this framework throughout the following sections, listing some of the most important lessons I learned about success and failure throughout my career.

Lesson #1: Technology Stacks Define Employability

This lesson is more of a blind spot for project managers and HR leaders than for software developers.

The advice of “Learn or Earn” works when you are starting in your career, but it morphs to “Learn or Don’t Earn” as you progress.

Within the last 40 years, the software industry went from co-dependent relationships of job-for-life using a few on-prem technology stacks to one of impossibly short tenures, leading employers to reconsider “job hopping” as a negative trait.

As a result, successful employees used to lifelong tenures now have to anticipate technology trends and grapple with the reality of becoming obsolete while standing still.

Note I am not passing a personal judgment on any technology. I am just reflecting on hiring practices in general, where employers increasingly use automation to match candidate resumes to opportunities.

HR folks may point to periodic formal training to stay current with trends while working with aging technology. However, there is a realization among software developers and employers that skills not practiced on the job dissipate quickly between forgetfulness and irrelevance.

As an example, when an employer needs someone with five years of experience in React, their vetting process will most certainly discard a resume showcasing a long experience in JavaServer Faces even if it contains a bullet point about a React class taken a couple of years ago. Conversely, any company using JavaServer Faces as their primary UI framework today will most likely struggle to attract new talent to their fold.

For those reasons, and acknowledging this may be a difficult realization for employers: From a continued employability perspective, a product technology stack is the most crucial factor for employees.

Yes, a bad boss can make one quit, but even the best of bosses will struggle to recruit people to work with a technology stack with dwindling demand in the job market.

While I understand product managers and organizations not being keen on rearchitecting or rewriting portions of the technology stack for employability reasons, they should consider a periodic conversation with HR leaders, trying to understand what effect their projects may have on the future employability of their staff.

Lesson #2: Three Years as the Period of Obsolescence

Technology stacks may stay relevant for a long time. Look no further than the enduring staying power of programming languages like COBOL or hardware technology like mainframes.

However, not all technology stacks stay relevant.

For everything else not powering the business core of the modern world, three years is a magic period where new technology can suddenly fall out of favor with the technical community.

If you are deciding on the next step of your career or the next hot technology to learn, skip the temptation of chasing what is popular today and look at what is trending towards popularity within the next three years.

Note that "three years" is primarily an empirical observation.

To illustrate that observation, let's start with the progression in the adoption of programming languages for the past decade, measured indirectly by GitHub's analysis of its user activity:

Why use programming languages as a proxy for technology stacks? Because the adoption of programming languages correlates with shifts in the technology landscape, such as Ruby’s fortunes being tied to the popular Rails framework, COBOL adoption relying exclusively on the mainframe ecosystem, and C# being the programming language of choice for the .Net platform.

In three years, between 2015 and 2018, the Ruby programming language fell sharply through the rankings, from spot number 5 to number 10, fading behind Typescript’s growing popularity as a superset of the reigning champion in programming language’s popularity: JavaScript.

Incidentally, in an equal span of 3 years, between 2017 and 2020, Typescript sky-rocketed from number 10 to number 4.

And look at what happens to PHP, a stalwart of employability holding out against the proliferation of web-development frameworks all the way to 2019, when it started dropping quickly through the ranks from number #4 to #7 by 2022.

If you were starting your career today, mastering any programming language that is not mainstream within your field of interest is a liability for future employment prospects.

By “mainstream,” I mean choices such as Javascript for front-end development, Python for machine learning, and Go for systems development. You may disagree with my specific examples — for example, choosing Rust over Go — but the principle remains.

If you prioritize employability, find the moving center of the mainstream and aim for it.

The following section will cover the risk of overshooting that target.

Lesson #3: Watch Out for That Bleeding Edge

Anyone who regularly rode a bus has learned to make a critical judgment call.

That call happens when you are late walking up to the stop and can see from a distance that the bus is already loading the passengers. In that moment, you must decide whether to sprint toward the stop or wait for the next bus.

You also learn that chasing an already moving bus is rarely worth it.

While looking at future trends, keeping an eye on employability over hype is essential. I don’t mean people should ignore new trends — jumping on them can be rewarding in itself — but I would not advise people to go all in into those trends without a hedge on the “Growth and Development” layer being advanced in other ways, such as high-visibility or leadership positions.

Be advised that there is a lot of course-correcting and damage control to manage that edge while the users try to build worlds atop the moving blade. In other words, work-life balance tends to suffer.

My cautionary tale here comes from first-hand experience during the cut-throat years between 2013 and 2015 — another three-year span — where Cloud Foundry and Docker initially defined and dominated the container orchestration world.

My employer teamed up with the creator of Cloud Foundry and launched its public offering in early 2014. Known as Bluemix, that offering quickly became the world’s largest Cloud Foundry deployment, with over one million users. By many accounts, it was an unmistakable success.

I joined that project from the onset as its production engineering lead, coming straight from a role as the technical leader for three software development teams. It was a wide lateral jump, so it was also a big gamble.

There was one planet-scale obstacle in my jumping arc.

Later that year, our company also made a parallel investment in a new offering based on an open-source project with a funny Greek name. Google released Kubernetes early in 2014 as an orchestrator of Cloud workloads. It had the pedigree of underpinning the mind-bending scale of Google’s own workloads.

With a billion-dollar investment and an all-star line-up of talent and developer advocates behind Kubernetes, the competition was over before it even started.

In the relatively short span of three years, Cloud Foundry and Docker Enterprise had lost all momentum in new deployments. Existing customers spent years grappling with the costly reality of re-platforming their workloads, retooling their CI/CD pipelines, and retraining their employees into a new world of Kubernetes clusters.

For me, it was not a wrong career move in absolute terms. I met people from different backgrounds, and the grueling demands of the on-call schedule for something at that scale meant a considerable compensation increase. And even after all these years, I am convinced that every software developer should rotate through operations for a few years.

Still, by the end of that transition, my Cloud Foundry skills were worthless in the job market.

Lesson #4: Beware the Highly Specialized Domain Projects

This lesson comes from the project I joined immediately after my Cloud Foundry stint. It is also the project mentioned earlier in this story, where we struggled to fill out a software development position.

The product was a hosted offering for healthcare providers in the oncology field. From a technology perspective, there was a strong focus on bioinformatics and natural language processing.

The most significant appeal for me was the opportunity to work with lab scientists and actual medical doctors. Applying NLP to help oncologists find alternative, life-saving care for cancer patients fulfilled the “Meaning and Purpose” layer in ways that no other project in my career ever had.

However, the technology stack was entering the three-year period where it would quickly fade behind the times, especially for a SaaS offering.

• Hosting? Everything ran on VMs. There was not a container in sight.
• Programming language? Java.
• Persistence layer? DB2.
• UI? AngularJS.
• Task management and source-code control? Jazz.
• CI? Ant and Jenkins.
• CD? Largely manual, with multi-hour planned maintenance windows.

With the unforgivable exception of using the aging Ant framework over Maven in the CI layer, everything else would have been a solid choice a couple of years before the project was commissioned.

In short, the choices were perfectly sensible, well-executed, and cost-effective. Decisions to stick with stack components were sound from a business perspective, but as an experienced software developer, this stack was the highway to skills obsolescence.

A small yet significant example is the usage of the Jazz source-code control. The year I joined the project was also the year our company formed its strategic partnership with GitHub, paving the way to replace Jazz with GitHub repositories for internal projects. Our project never migrated from Jazz because it made no business sense to forklift all code and build processes to a new source code control system, let alone retrain all developers to use Git.

Still, from the perspective of software developers, the “employability” value of Jazz skills trended toward irrelevance in the ensuing years, almost in inverse proportion to the value of Git, GitHub, and GitLab skills.

Moving to a more relevant example, querying all the alterations in a genetic sample stored in a database did not require a software developer to understand cancer biology. It required good communication with the actual cancer biologists on the team, explaining the immediate table structure representing the data and figuring out which attributes on a table represented those unique alterations.

At the same time, the database world was witnessing the expansion of job market demand for skills in MongoDB — the poster child of NoSQL databases — Cassandra, Neo4J, Apache Spark, and Hadoop.

And while Java was — and remains — a desirable skill in the job market, people working with the project backend watched Python slowly take over as the primary programming language for anything related to machine learning and natural language processing. If that was not sufficient, Facebook and Google released PyTorch and TensorFlow in rapid succession, increasing the pace of skills obsolescence for those working on machine learning problems in Java.

In a couple of years, our technology stack went from sensible to something no one would consider for a new project in the same domain.

Ultimately, that is why we struggled to find new software developers while, at the same time, the sciences team managed to recruit increasingly more impressive talent.

Even without this story’s career framework clear in my head at the time, I would be open with candidates about the trade-offs between the unique opportunity of working in a specialized domain and the negative impact on their long-term growth and development.

Some candidates had a specific interest in the bioinformatics and NLP components of the project. Since we were not hiring for those areas, I was equally open about eventual work on those components being strictly dependent on changes to their job roles and, therefore, not guaranteed.

I still cherish the experience, but here is the painful reality for software developers working with undifferentiated technology in a highly specialized domain: domain experts are essential; software developers are not.

That reality would be punctuated three years into the project when it became clear that demand would not translate to meaningful financial results.

The team was considerably downsized to stretch its budget runway while hoping for a turnaround. Anyone not working directly with bioinformatics or machine learning — most of its career software developers — was asked to find other opportunities with the company, myself included.

Lesson #5: There Is Always Time Until There Isn’t

Back to the bus-riding analogy, there is something worse than missing a bus: boarding the wrong one.

My cautionary tale in this story is that even though I was clear in my objective of shifting to Cloud-based software development for several years, in between a failed start with Cloud Foundry and an attractive side-step into the world of life sciences, six years went by in what today feels like the blink of an eye.

For someone mid-career, six years is a dangerously long detour.

It took me a new job and another three years to catch the Kubernetes bus and take a seat, becoming skilled on the platform and acknowledged as someone who could advise other developers and system administrators.

Nine years had passed.

For someone starting fresh from college today, six years traveling in the wrong direction is a career-defining interval.

Many will point out that the “wrong direction” is relative and that having a diverse background before starting a software development role later in life is possible, even desirable.

I do not disagree. In fact, one of my favorite software developers has a degree in Literature and spent over a decade playing various musical instruments at shows before starting a successful career in software development.

However, once one gets in the groove of a career path in tech, indulging in career detours and missing consequential technology shifts can often derail their progress for quite a while.

Yes, we always have time, but career steps are measured in years, and the margin for recovery decreases with each passing year.

Conclusion

My best career advice to software developer professionals, well, any professional, is to continuously monitor their career progression against the layers of their “Hierarchy of Career Priorities.”

There are no wrong career paths, but there are certainly careless steps. Something as simple as placing an “up” or “down” arrow next to each layer in the pyramid can go a long way toward helping one make the best decision about staying in their current role or moving to a new one.

With awareness, one may optimize for one or more layers, such as choosing a higher compensation over a more interesting job or sacrificing their work-life balance for faster progress toward their life goals.

One may also skip hierarchy layers altogether, such as deciding to work for a dream non-profit with virtually no compensation or betting on a startup gig with compensation tied to the uncertain guarantee of the company’s success.

However, those imbalances tend to catch up with the natural order of priorities in the pyramid. The imbalances manifest in different ways, such as depression induced by lack of meaning on the job, burnout from long stretches of 80-hour work weeks, or deteriorating employability due to working with obsolete technology stacks.

And dream jobs may not be that satisfying when one is constantly awakened from the dream by unexpected bills or stressing out with personal finances between irregular paychecks.

No matter the chosen path, having a framework to evaluate your career is your best tool for making judgment calls, whether extending a bet on a risky choice or bailing out when a job becomes a potential liability to your career prospects.

That framework is also invaluable for companies gauging their attractiveness to prospective candidates, especially when technologies can shift faster than projects can respond to those changes.

And finally, keep aiming for purpose and meaning.

Be successful.

We can describe most problems as “Something is not working like it should.”

When things don’t work as expected, it is time to practice some form of problem-solving. For example, if we want to restore a system to a running state, we can try probing methods such as Root cause analysis or Five Whys. If the problem involves challenging requirements while developing a new product, we may favor an exploratory method such as Design Thinking.

And when it all fails and a problem exceeds our skills and resources, it may be time for the well-known problem-solving method of help-seeking.

Asking for help is a familiar routine in a virtual world with an infinite supply of “somethings” that can fail or misbehave at any time, creating familiar challenges all around:

• Ask for help too early, and you risk wasting someone else’s time.
• Ask for help too late, and you risk missing an important delivery date.

It is in between those extremes that most organizations struggle to find balance.

Recognizing when to seek help and how to be ready to receive that help is the foundation of the help-seeking social contract, minimizing wasted time and maximizing collective progress.

Sometimes, It Does Hurt to Ask

One may argue that when the delivery dates are slipping or the SLA service credits are flying out of the window, it is always time to seek help. Right?

Not quite. We need to do some homework first.

Here is an example:

Developer: “@channel My nginx server isn’t working. What could it be?”

Slack UI: “Uh, Chief, do you really want to alert 3284 people across 11 timezones?”

Developer: “Yes, Mr. Bot. This is the darkest hour. Wake them up. The whole lot.” <clicks “Send”>

Twelve replies into a hailstorm of reprimands and lectures from the channel constabulary, someone finally breaks through the virtual mob: “Is it accepting connections?”

The hapless developer answers “No,” confused by the question’s relevance because “I already said it isn’t working!” He awaits what must surely be a final solution in the following answer, but nothing happens. The helpful bystander is seemingly gone for the day. But fret not; the developer messages his local senior developer and repeats the original question.

With all forward momentum converted into heat, wear, and tear, the developer awaits the rescue. Sitting by the side of the virtual road — a terrible place to be — the developer rationalizes his situation, reminiscing about the advice received earlier in his career:

• “Don’t spend too much time on it. If you get stuck for more than an hour, reach out.”
• “Don’t be afraid to ask for help. We are here to support each other.”

Those are unquestionable truths, but they are of the guardrail kind. They focus on what should not happen but don’t guide people on how much time is too much nor set the bar for a question being perceived as too basic.

What our hypothetical-but-all-too-real developer needs are clear expectations and usable guidance that make problem resolution more accessible and attainable.

That is this story’s topic, divided into three simple steps.

Step #1: Trust the Crowd. The Answers Are Out There

There was a time when the developers building the product or the system were the only source of information.

Getting things to work was challenging enough, and making documents user or task-friendly was an afterthought. Access to other users was limited, as was the access to the product’s source code. Finding information about the system and existing problems was protracted and meticulous work.

That was a long time ago.

Except for bespoke systems built in-house by a handful of people, everything else nowadays is behind well-documented service APIs or available as open-sourced code. Access to information about products, libraries, and services in a modern Cloud environment is a solved problem. We can always gripe about some blind spot here and there, but the junior engineer of today is two web searches away from being more effective in solving their problems than a senior engineer of twenty years ago.

Beyond factual information answering simple “what is” and “where can I find” questions, public forums are treasure troves of information.

For all the complaints about the toxicity in web forums, muscle through the noise and you can find threads filled with valuable discussions. There, you will discover seemingly random people who post the correct answers to questions and debate the nuances of why incorrect answers are wrong and the range of situations where solutions will work. Hang around the comments section for an extra moment or two, and you can pick up on many other clues and related links to help solve the problem.

AI-powered bots like ChatGPT can also answer questions. However, I think the sanitized format of the answers does not offer the same rich learning experience of following one’s nose through web search results. After all, isolating positive signals from background noise is an invaluable problem-identification skill.

Step #2: Keep Moving. Help Will Follow

In nature, being stuck is a precursor to death. Conversely, in the virtual world, being stuck is a precursor to virtual death.

Lack of progress is depressing and demoralizing. It kills the spirit instead of the body. Yielding control over your progress to someone who can help restore that progress is a last resort to be used sparingly.

I am not writing this to shame people into not asking for help, but there are ways of asking for help without stopping while waiting for help.

“What if I really don’t know what to do? Should I be afraid to ask for help?”

No, you shouldn’t be afraid. The time to ask for help may come and will be covered in the next section, but you probably can do a couple more things to keep moving. How many times have you asked a question only to think of the answer a minute later? That is your brain working through the intricate lattices of semantic networks and spreading activation.

As you shift your focus from solving the problem to describing the problem, your brain fetches different schemas to convey the information from a different angle. Those different perspectives often give you a clear line of sight to the problem’s resolution.

“So, should I ask my question or not?”

While you can keep working the problem in your head talking to a Feynman’s rubber duck, having access to a large team channel or Internet forum offers a powerful alternative: sharing the problem and working steps with others while you are still trying to solve the issue.

To quote comedian and philosopher Chris Rock:

“I used to have horrible cars that would always end up broken down on the highway. When I tried to flag someone down, nobody stopped. But if I pushed my own car, other drivers would get out and push with me. If you want help, help yourself.”

You can bring that lesson to software development and systems operations by framing your problem in a public thread like this:

“Hey, I am seeing the error message “XYZ” in this system. I am still working through it and will be posting my findings as a thread. Chime in if you have seen this message.”

Assuming you took the first step to heart and the solution to your problem is not the first result of a web search, starting such a thread in plain view of channel participants brings in multiple benefits:

1. You can keep moving because you have not yet exhausted your resources and options.
2. People can help you in their own time, so you are not interrupting their flow.
3. Other people interested in the topic can follow the thread.
4. Reframing the problem as something that can be shared with others can help you solve the problem

If later you find the solution and the question is not too basic after all, then you can post the answer and other people hitting that problem in the future can find a ready-made solution.

With this step, you are not flagging cars down the road; you are stopping to help others push their broken cars. And because this is a virtual world with little distinction between footprints and the body leaving them behind, you may be helping others without knowing.

Personally, I am somewhat spoiled by working in a large company with large Slack channels. My searches in our internal workspaces turn up a helpful answer 95% of the time when I hit a new problem. They turn up a working solution in almost 80% of the cases. And for the remaining 20%, taking the failed search to the Internet brings the search-to-solution ratio to almost 100%.

One can still be highly dependent on others to complete their work, but one doesn’t need to be blocked waiting for help.

If you worked through the first two steps and still haven’t found a solution, it is time to ask someone to commit their time and resources to assist with the resolution. The following section shows how one can maximize their odds and minimize their losses (of time) by gathering a few facts ahead of time.

Step #3: Gather Your Facts. It is Show (Me) Time

As that hypothetical developer in the introduction should have realized, being ready to request assistance can make the difference between receiving a solution or another question.

A senior developer may be able to bridge the gap between what is provided and what is missing, but that is a wasteful process where precious time is spent on gathering facts that do not require any domain expertise. Just like we shouldn’t use a torque wrench as a breaker bar or a scalpel to open a plastic bag, we shouldn’t use an expert’s time to gather basic facts about the problem.

“OK. How much information about the problem is information enough?”

This is a question of context.

When you pose a problem to a colleague in the same project, you probably share a lot of contextual information about the project and any new issue — “It is that same problem we saw last week.” Conversely, a stranger maintaining an open-source project will need much more context about a problem you may face with their project. That is why reporting an issue with something like Kubernetes.io requires filling out a relatively thorough form template with several fields.

Following is a checklist I learned from a Critical Thinking coach many years ago, a couple of years into my first assignment as a team lead. I thought I was good at quantifying and qualifying problems until I met this coach, a retired manager in his mid-60s. He made a living by solving problems ranging from homeowner association disputes to CEO succession plans.

The “What?”

• What is the expected behavior of the product?
• What is the observed behavior in the product?

The “Where?”

• Does the reported problem happen to all units of the product?
• Does the reported problem affect the entire product or just portions of it? If so, describe the parts of the product.
• Does the reported problem happen in all locations where the product is used?
• Does the reported problem occur in combination with other issues?

The “When?”

• When did the problem start? If you don’t know, make it clear you don’t know and state when you first observed it.
• What is the frequency? Continuous, cyclic, or random?
• Is the problem specific to the phase in the product life cycle?
• Is the symptom stable or worsening?

Returning to the initial example in this article, when that anxious developer asked, “My nginx server isn’t working. What could it be?”, can you spot everything that is wrong with it?

In other words, could you expand the description to a point where an expert without familiarity with that system would not have to ask clarifying questions?

Going through the checklist, here are a few examples of questions I would be asking about the problem:

• “What is the expected behavior of the product?” — Obviously, we want the nginx server to be working, which entails … what? An nginx server can serve diverse purposes, such as reverse proxying, load balancing, and caching. There is room for more precision here, but let’s enter a state of “suspended curiosity” and assume the developer means every possible function available on a typical nginx configuration.
• “What is the observed behavior in the product?” — “Not working” is not a behavior. Is the server not starting? Is the server crashing after starting? Is the server rejecting connections? Is the server routing connections to the wrong handler? Is the server hanging while processing requests?
• “Does the reported problem happen to all units of the product?” — Are there other nginx servers with a similar configuration working? For example, is this problem only happening on a single instance in a pool of nginx servers where all other nginx servers are working without issue?
• “Does the reported problem affect the entire product or just portions of it? If so, describe the parts of the product.” This question goes back to the entire range of functions available on an nginx server. Is there any function that is working? For example, requests to a specific URL may return the expected results, but requests to other URLs hang indefinitely.
• “Does the reported problem happen in all locations where the product is used?” If this nginx server template is deployed to multiple Cloud regions, is the problem localized to one or more regions but not all?
• “When did the problem start?” Was it on a specific day where you have evidence of the functioning working on the day before? Was it immediately after a configuration change was applied to the environment? Note that this question may be the most difficult to answer because you may not have good observability practices placing the exact moment where the problem starts. In that case, report the first time you or someone noticed the problem.
• “What is the frequency?” Are all requests failing, or are some of them going through? If some of them are going through, is there any pattern, such as failures happening continuously for a couple of hours and then working without issue for a few days?
• “Is the problem specific to the phase in the product life cycle?” Does the problem only happen after the server was started for the first time after rebooting their hosting VM? Is the problem only happening after the server has been up for a week?
• “Is the symptom stable or worsening?” If the problem happens randomly, is the average rate of request failures going up? If the issue only occurs after the server has been up for several days, is that number of days coming down?

Bonus Steps: The Team Side

The previous steps covered the individual responsibility aspect of tackling situations where we feel like we could use a hand.

Flip sides on the help-seeking contract, and there are also measures that the leadership team can put in place, from managers to senior individual contributors:

1. Incorporate some version of the problem identification steps in the previous sections into regular onboarding and training sessions, setting the tone and the bar for all team members.
2. Favor open-source libraries and components over home-grown bespoke systems. If you want individual team members to lean into crowd-sourced assistance, you need to widen that coverage by using the same technologies and speaking the same language as the crowd.
3. Promote and incentivize a culture of team collaboration. Push people toward communication in online forums and problem-tracking systems instead of private messages.
4. Organize regular public post-mortem events where software developers and operations people review interesting bugs and outages. An engaging puzzle is at the heart of every bug and outage — just look at these.

A culture of tackling problem identification and dissection head-on is a culture of agency, resiliency, and self-sufficiency. Mutual assistance is still part of the social contract, but it is evenly distributed across offline and online interactions so that people can keep moving — staying engaged and motivated — even if they could use an extra hand.

Conclusion

The simple steps in this article can help professionals across all experience levels raise their individual problem-solving and problem-reporting skills. With some practice, people struggling with seemingly trivial issues will get stuck less frequently, keeping their motivation and momentum through tasks that felt impossible.

On the other side of the social contract, senior engineers working on a team that consistently follows those steps spend less time addressing basic questions and less energy trying to fill the gaps in questions and requests. With more disposable time comes a renewed focus on meaningful interactions and real progress.

For organizations, the newfound self-sufficiency and precision become gains in productivity, quality, and morale.

The best part is that anyone can practice these steps in a matter of days and experience the benefits immediately in their next few assignments.

Whether you are looking for help too often or you want to help others in more meaningful ways, consider the problem solved.

Kubenstein is a side-project named after a play on the words “Kubernetes” and “Frankenstein.”

I wrote it to experiment beyond the more straightforward use cases for GPT-powered assistance to Kubernetes administrators, such as in querying cluster resources or in interpreting errors in cluster messages.

You can see examples of those scenarios in CLI extensions like k8sgpt project and UI integrations like Robustas’ Slack bot.

These are natural and valuable additions to existing tools, but Kubenstein goes beyond the sensible and the reasonable.

The following image shows an excerpt of a conversation between the Kubenstein application and its AI “brain.” The “lines” from Kubenstein back to the AI are the output of the commands suggested by the AI when executed against a target cluster.

Beyond Language. Intelligence?

LLMs can do a great job of natural language processing, deciphering the written commands from a system administrator and summarizing the collective human knowledge about an error message.

However, beyond language processing, I wanted to explore the possibilities of AIs following cognitive scripts:

…human behaviour largely falls into patterns called “scripts” because they function the way a written script does, by providing a program for action.

I wondered whether I could tell Kubenstein how I perform a particular activity and let it attempt to replicate it. For that, I chose a familiar cognitive script for Kubernetes administrators: assessing the cluster health during a troubleshooting session.

Anyone assessing the health of a cluster has their preferred way of going about it. Still, these assessments more or less involve serially inspecting layers of components and looking for errors.

Along the way, when they bump into errors, they start looking individually into each error for clues. It is never a fixed recipe because one never knows what they are about to find. Still, the outline, the cognitive script, shares many commonalities.

So I started to think:

“How would an AI perform as the driver of a troubleshooting session?”

That is right; I didn’t want assistance to be understood more easily or to analyze error messages more efficiently. I wanted something that could follow my mental process and do the work.

Chat over Prompts

The core goal for Kubenstein is to perform administrative actions without depending on human intervention. For the specific use case at hand, I wanted it to be able to run a troubleshooting session while following my cognitive script for that activity.

I did not want to throw errors at the AI to get guidance. I didn’t want to be in the middle of the interaction. I wanted Kubenstein to interface with the cluster on its own.

Kubenstein runs as a standalone Python application on a terminal. It assumes the human administrator has already logged in to the cluster using the kubectl CLI and that the CLI is available in the PATH environment of the terminal.

It also requires a connection to an AI, such as ChatGPT, to start a chat session with my cognitive script of the task.

During a troubleshooting session, Kubenstein iteratively requests the next input command from the AI, executes it against the target cluster, and then adds both the AI input and the command output to an array of messages representing the chat.

That process is repeated, with Kubenstein sending the ever-growing array of messages to the AI so that the AI understands the whole session context. The process ends once the AI responds with a message indicating it has completed the cognitive script.

In that sense, Kubenstein deviates from the current Kubernetes assistance approaches by mediating a conversation between the AI and the cluster.

Kubenstein does not attempt to make you faster while troubleshooting a problem; it tries to be you.

The starting point of that conversation is instructing the AI to act like the system administrator and informing it about the goal of assessing the health of a cluster, providing an outline of how to go about the whole process, and letting it run through the instructions.

For example, the initial assessment prompt contains phrases like:

You are a Kubernetes administrator asked to validate whether a cluster is running properly.

And blocks of instructions like this:

I want you to help me go through a checklist called “cluster health” step-by-step.

- Nodes
- Storage
- Routes
- Network traffic not getting blocked
- Pods
- Events

More crucially, remember this is a chat session between the AI and the cluster; this is the most important instruction to the AI:

It is extremely important that you give me a single command at a time and wait for me to execute the command and give you the output.

The prompt proceeds to clarify what should happen once the AI encounters a problem:

For each item in the checklist “cluster health”:

- Wait until you can process the output of a command before deciding what the next best command is.

- If the output indicates an error in the cluster, add that error to an error list named “error list” and proceed with the checks on the checklist “cluster health.”

- Once you complete giving me commands for the checklist “cluster health,” we should start troubleshooting the list named “error list.”

You can read an example of an entire assessment session here.

Learning As It Goes

There is nothing in Kubenstein to try and train it about what to expect and how to react to what it finds, just like I wouldn’t know what to expect while troubleshooting a cluster.

Personally, if a command shows a problem in the cluster, I adjust to that response before typing the next command. If I make a mistake while typing a command and get an error message, I modify the command and iterate until I get it right.

So does Kubenstein.

I must admit there were several moments where I doubted a generative AI could make sense of a long and intricate set of instructions, let alone take the contents of a chat session into account in making corrections while keeping track of its larger goal.

To my surprise and encouragement, not only could the AI parse and go through the whole script, but the mistakes and course corrections made Kubenstein look surprisingly human.

I know; I wrote how AI outputs never look truly human. Still, in the context of a troubleshooting session, where a chat session transcript is filled with technical terms, input commands, and CLI outputs, the most significant indication I was not reading the output from a session driven by a human administrator was an unflappable determination in getting things right.

To illustrate that point, while Kubenstein has no specific Kubernetes training, it has a few boundaries about the conversation length, such as AIs limiting the number of tokens in API calls and the fact their providers charge calls by number of tokens. Due to those limits, Kubenstein avoids adding command outputs with more than 500 tokens to the chat session, instead adding the following message into the conversation:

The command you gave me returns too much output.

- Give me a version that filters out more content, such as selecting a specific namespace or returning resources that are not in a state normally associated with a healthy resource.

- If asking for logs, try and apply filters with the “grep” utility followed by the “tail” utility to return only the most recent entries.

- If you don’t have alternatives, then proceed to the next check.

Whenever presented with a response indicating a problem with the command, the AI returned a response with an apology and a new attempt at a command, like this:

Apologies for the inconvenience. Let’s try a more focused command to check the status of routes.

To filter the routes that are not in the “Accepted” state, please run the following command:

kubectl get routes --all-namespaces --field-selector 'status.state!=Accepted'

Keen Kubernetes administrators will notice this command has a bad field selector, which is not a problem for Kubenstein, as it feeds the resulting error message back into the chat session, prompting the AI to return this new attempt:

Apologies for the incorrect command.

To check the status of all routes across all namespaces, please run the following command to retrieve the Routes resources:

kubectl get routes --all-namespaces

Please provide me with the output of the command.

And while these self-corrections cost precious context length — and money — it is still somewhat disconcerting to witness an AI adjusting to a live situation without hand-coded error handling or training outside the chat context.

You can see a formatted example of a completed session here.

A New Programming Paradigm

Deferring most business logic to an online third party is a significant leap from traditional software development.

The visible part is evident, with plain-text cognitive scripts replacing code branches with clear execution paths. The consequences need some spelling out.

The traditional application is much shorter (the entire chat application for Kubenstein is around 200 lines of code). Still, you have to spend time encoding plain English scripts.

Iterating over cognitive scripts is initially more productive than writing traditional code, skipping the encoding of mental structures into a syntax-heavy programming language. However, those cognitive scripts do not have the precision of an actual programming language, so you have to account for being misunderstood and have some minor reinforcement prompts in place.

I also noticed inconsistency in the AI responses, where it suddenly started to go off script between identical conversations, requiring adjustments to the initial prompt and those smaller reinforcement prompts to remind the AI of the parameters of the initial task. For example, while I instructed the AI never to return commands with placeholders, there are a few moments where it does it anyway, so Kubenstein “reminds” the AI about its expectation.

Those short-loop runs from inside your favorite code editor? You no longer can step into the most crucial part of what the code is doing, and each run costs real dollars. I mean, fractions of real dollars, with the typical troubleshooting session through Kubenstein costing about US$0.05.

While it is easy to shrug off the prospect of running up your cloud bill by a handful of dollars per month, a single coding mistake involving an unbounded loop cost me US$2 in 30 minutes, a money drain only interrupted by an email from the provider alerting me about my account crossing its soft-limit for the month.

And even if you have a good handle on such mistakes and optimize your local runs, there is still the fact you are outsourcing your application “brain” to a third party and have little recourse if the vendor raises prices beyond the envelope of your business case. As one such example, OpenAI’s GPT-4 cost per 1K tokens is 10–20x more expensive than GPT-3.5 Turbo.

The risk mitigation for that outsourcing is simple and costly: validate that other competing AIs can process the same conversation and be ready to switch providers immediately. While at it, remember that not all AIs can process cognitive scripts. For example, I could not replicate Kubenstein using Google’s Bard AI.

And speaking of third-party providers, when the full potential of AI means cutting off humans from the loop, a fully automated Kubenstein can potentially damage the target cluster or leak sensitive information to the AI provider. Until some form of trust is developed in the system, that means adding human supervision into the loop.

Conversely, you get functional coverage you could only dream of for a couple hundred lines of code. The application can handle use cases you have not considered because it can adapt to error messages you have not seen before.

To Err Less Is Human. The Next Steps

Part of me expected Kubenstein to be a flop, with the AI (GPT-3.5 Turbo) hallucinating nonsense within the first few minutes.

Before committing to writing an application, early sessions using the AI chat interface surprised me with its ability to recognize and follow complex cognitive scripts.

The early success gave way to slight deviations from localized instructions, such as providing me commands with placeholders or multiple commands in one response.

Also, any Kubernetes administrator looking at the example transcript mentioned in this story will spot consequential mistakes from Kubenstein in narrowing down the assessment to the default namespace and concluding the cluster was healthy without inspecting the full scope of what it had been asked to.

While I like the idea of not training the AI and not giving it examples, not to mention being excited about birthing a semi-competent system administrator with a mere 200 lines of code, it is clear that Kubenstein can benefit from more examples (via Retrieval Augmented Generation perhaps?) and that the code base should grow somewhat to split and merge conversations to minimize the traffic — and attendant cost — between the application and the AI.

That cost aspect, both in time spent coding and money spent “thinking,” will probably be ever-present while creating new iterations of Kubenstein. I plan on keeping notes and writing more about it along the way.

Kubenstein, a GPT-powered Kubernetes Administrator was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

The pandemic is over and it is time to return to normal.

And in the business world, that normal includes the traditional office.

But why?

Step away from conspiracy theories of power plays and real estate valuations, and you will find two simple reasons rooted in psychology and neuroscience:

1. When things become hard to navigate, we tend to seek familiar surroundings.
2. We don’t remember things the way they were. We remember the way we felt about those things.

What can be more challenging to navigate than improvised work arrangements imposed by death, panic, and lockdowns? Which period brings out the worst memories of recent times?

None of this means returning to a point in space will bring back those years, but like a crashed system, it is only human to revisit the past, hoping to find a stable snapshot and restore what was lost.

Speaking from my corner of the corporate world, in software development, we adapted better than most to the partial lockdowns, not the least because we normally excel in that twilight between the virtual and the real.

It was still improvised, and it was still a pandemic.

The flexibility of working from home paled compared to everything else lost between 2020 and 2022. Beyond the obvious death toll, there were also layoffs, school years, and sanity.

Then, 2022 brought about a rollercoaster of irrational exuberance in hiring followed by more layoffs. At some point, anything pre-2020 started to look like a golden era.

There may be sound reasons for returning to traditional offices. Still, some arguments around engagement, networking, and mentorship are missing significant cultural and technological shifts underway before COVID-19 struck.

Before getting into those, let’s start with the giant elephant in the conference room.

Slack: We Hardly Knew Ye

The pandemic kickstarted an unprecedented experiment in the workforce worldwide, generating something I did not think we would see back in our (professional) lifetimes: slack.

No, not the collaboration platform; we saw plenty of that. I mean “slack” in terms of unassigned resource time in a project.

Suddenly, people were not commuting anymore. Taking the United States (of America, for those reading from abroad) as a reference, where the average one-way commute is a little shy of 30 minutes, and assuming ten trips over five days a week for in-office work, we had an extra five hours in our hands every week.

Five hours no longer spent on something scientifically proven to be a significant stressor on daily routines. Gone.

Shortly after the initial productivity shock of lockdowns, we realized the slack went beyond those five hours because we could shuffle our routine around flow states.

Once school closures stopped being the norm, a share of the WFH population also experienced a newfound ability to outperform on tasks that required deep focus and concentration: Fewer unplanned interruptions!

More work done. More slack.

And before we write off commute time as a shrug-worthy chore that can be filled with audiobooks and podcasts, we should get into the math of disposable versus usable time.

It is easy to rationalize the average five hours of commuting as a small percentage of a typical week, with 168 hours spread across seven 24-hour days, amounting to a mere 3% (5/168) of that total.

However, take weekends (48 hours,) work week sleep (40 hours,) and working hours (40?) from the denominator, and those five hours suddenly become 12.5% (5/40) of the total.

Continue that exercise, removing other fixed costs attending to unavoidable personal chores, reusing some data from the U.S. Bureau of Labor Statistics, and those hours start to represent anywhere from 20% to 50% of the leisure time for a working adult, depending on whether they have children and the children’s age.

So, while RTO may just be a return to a pre-pandemic normal, it is still a massive shock to present-day professional and personal routines.

Net: There are concrete reasons to try and extract the benefits from in-person collaboration from fewer days in the office. Credit to most RTO policies, they seem to acknowledge that fact with hybrid arrangements of 2–3 days per week in the office.

Uneven (Virtual) Reality

In the tech world, for better or worse, we are conditioned not to hold on to processes or tools. Our past is an undesirable and uncompetitive space, littered with underpowered hardware, slow connections, and gray walls.

In tech, there is no normal. There is no familiar.

So, why return to the old ways? Why not work from home — or away from the office — all the time?

We often get lost debating issues of convenience and flexibility — because they still matter — but we don’t arrive anywhere because reality is not evenly distributed.

Sure, at least personally, the coffee at home is better, the hours more flexible, and the commute infinitely shorter.

That is not the case for everyone.

For every person fortunate enough to be able to afford a dedicated home office unperturbed by kids in grade school, there may be someone taking a phone call ensconced in a broom closet while young children pay loud tribute to their Summer break.

For every seasoned professional who can work more productively without interacting with others in person, there is a junior engineer who could benefit from polishing their social bonafides and making essential connections that will help them later in their careers.

We will not find common ground in the anecdotes. They line up dangerously close to demographic boundaries, where here be dragons and significantly different outlooks in life.

From the sidelines, we also have to contend with a growing mental health crisis, where employers and employees need to be better prepared to navigate those conversations.

I am threading a needle of bothsideism here, but that is the whole point.

This debate will not be won with a decisive data-driven knockout, mainly because the little data we have reveals a more nuanced picture, with some segments of the office population reporting improved mental health post-RTO.

So, the current point in time is inconclusive, but as the following section shows, those demographics are pointing in a clear direction.

The (Generational) Point of No Return

RTO is a centuries-old cyclical phenomenon.

In the long run, it is all about colocating the tools and materials required to do the work in a way that makes it possible and efficient, in that order.

Technology in software development moved in flash steps, steadily advancing through decades of exponential progress.

It is one thing to adapt to changes; it is another thing entirely different to experience the formative years of your career with a different reality.

In the early 80s, everything needed for software development was hugely expensive, large, and wired to the building — Say hello, fellow Gen Xers.

Fast forward to the early 2010s, and remote work became possible and reasonably seamless right before Gen Zers started joining the workforce.

By then, broadband Internet access was fast enough to make Cloud computing accessible from a home office, collaboration tools were much better, and teams had (somewhat) learned to cooperate across geographies and timezones.

Widespread video-based cooperation did not become an enterprise reality until the late 2010s, with Zoom and WebEx — They started much earlier, but the large-scale enterprise readiness came later.

Why the recap?

Because direction matters more than the stops along the way.

The software developers starting their careers in those early remote-ready years are now in their late 20s and early 30s. Statistically, that remote-first generation of tech workers is still a minority, around 20% of the market.

But it is not only a significant minority. It is a fast-growing group, both in size and influence.

Within a decade, that share of the workforce forged in a new reality of unfettered access to information will cross the majority threshold, including the C-suite.

That generation of business leaders will be uniquely adept to a remote-first environment of sophisticated collaboration tools and be socially optimized for efficient cross-geo, cross-timezone cooperation.

While I also worry about the long-term effects of human communication over two-dimensional small screens, the near future is already pointing at virtual life-sized 3D interactions.

Look forward just a little bit — years instead of decades — with the unlimited possibilities of reality augmentation powered by conversational AI, and in-person communication will inevitably become the less efficient, less fluid form of interaction.

Net: Observe and learn the communication patterns and preferences of Millennials and Gen Zers before making the case for the best balance between face-to-face and online communication.

Information Sharing Over Interactions

If you need complex information that only a colleague has in their head, face-to-face interaction and easy access to a whiteboard is the standard to beat.

One must step away from that ideal scenario and consider the reality of geographically distributed teams anchored around collaboration tools.

1. Essential information should not be in the heads of a single colleague. Knowledge hoarding, intentional or not, is a form of holding people’s productivity back, and organizations should optimize for forms of communication where that arrangement is short-lived.
2. In-person communication kills knowledge accumulation. Unless the person receiving the information proceeds to document what they learned, that means people in other offices will not have access to the privileged knowledge.
3. In-person communication excludes people in other time zones. No matter how hard you try, capturing an in-person session that explores the full range of expression of the in-person medium is virtually impossible.

We hear a lot about “remote first,” but what matters most is “online first.”

Before the Internet, we had to read manuals from cover to cover or struggle.

My first software development lab had a small library of technical books attached to it. The knowledge had to be in our heads, and RTFM was the law of the land.

Today?

I can hit Stack Overflow and get my answer in seconds. And it is not only the answer. Spending a few extra minutes reading the comments section — the true gold of the Internet — surfaces all sorts of alternative solutions, with the where and how they would work better or worse.

As a result, I can operate at a higher cognitive level. We all can. For lower-level information, we only need to fetch some keywords from those higher-level contexts and put them in a search field.

As a side note, for narrower questions where we need a snippet of code, there is always generative AI, such as ChatGPT. They may hallucinate a parameter or two but are borderline magical with Python code and CLIs.

Could you ask those questions to someone sitting next to you? Yes.

Will it get better business results? Absolutely not.

Net: Focusing on the quality of one interaction obscures the focus on the frequency, nature, and quality of all interactions.

Manage The Work First

People’s productivity ebb and flow with the nature of the project, assignment, personal issues, availability, and so many variables.

“But how can you tell whether someone is delivering results?”

Not by looking at them.

I have never been a people’s manager, but I have plenty of experience as a technical leader. In that sense, I am used to defining a technical solution and breaking it down into the tasks required to deliver it.

That breakdown also includes an assessment of the task’s complexity and the potential time it would take if assigned to someone experienced enough to take on that task.

Managing the work is my only recourse to keep things on track.

Is the work clear? Is it understood? Is it assigned to someone with enough technical skill to execute it? Is it being completed to the project standards? Does it meet customer requirements? Is it late?

These are the questions I can ask.

Does someone understand work better than others? Is the task owner taking longer than I would consider a reasonable time? Does their work environment allow them to focus on the work? Are they working on the task as much as I think they could?

These are questions I cannot ask. I eventually learned I was better off that way. Not being a people manager, a technical leader needs different tools to get the most productivity from a team.

Incidentally, the best productivity lesson I ever learned on the topic started with reading Peter Drucker’s “Management Challenges for the 21st Century” many years ago, from where I quote:

In the knowledge economy, everyone is a volunteer […] Accept the fact that we have to treat almost anybody as a volunteer.

While that may sound a little like letting everyone do whatever they want, or letting people not do anything at all, that mindset means you must define (and sometimes steer) the work toward things that attract people. It also means talking to people and asking them what interests them the most.

Personnel aspects can obviously ruin the business if left unattended, such as interpersonal conflicts, unmet career aspirations, life-changing events, compensation, and many others. Good people management still matters. And it matters more than ever if we will make the best out of hybrid office policies.

Net: When it comes to productivity — and acknowledging I am biased by my professional career choices — managing the work is often more effective than managing the people doing it.

Serendipity, You Say?

Serendipity in forming ideas is often cited as an advantage of in-office collaboration.

Supporters often tell stories of how people have randomly bumped into each other on the way out of a cafeteria or coffee shop, struck up a conversation, and came up with great ideas.

Ignoring the effects of personality traits in serendipity encounters, which may be experienced in very different ways across the office-dwelling population, the flaw in the in-office serendipity theory comes down to numbers.

Access to information (and people) today is not only better than it was three decades ago, it enables communication patterns that simply did not exist.

The breadth of knowledge around the cafeteria or coffee shop is infinitesimally smaller than what is available online.

That is not a diss toward my nearby colleagues; I also include my knowledge in that pool. It is just a fact that the online world has a much larger and more diverse population than you can find in a single building.

Watching a few key channels in something like a corporate Slack or Teams workspace will always be more effective in mining (and generating) ideas than hoping to bump into a similar spark from someone living within driving distance of the building.

And those online exchanges are not low-quality interactions. They are rich, augmented with in-context lookups, and filled with links and referrals.

Yes, meeting the people on the other side of those chats in person would be fantastic, but that is beside the point, as we would not have met them in the first place.

Most of my favorite professional colleagues live hundreds, often thousands of miles from my office — I am sure some of yours, too — Some live in different countries. And I may never meet them in person.

Net: Create conditions for collaboration first, actively encourage people to understand the difference between intention and luck, and then push people to look for serendipity in ways that work best for them. Then, periodically, find ways to gather those people in person, using group events and planned activities.

We Are Here. Make It Count

The current reality of how the workforce relates to the traditional office is mixed.

The future direction is not.

There is nothing wrong with wanting face-to-face, high-quality interactions, easy access to information, supporting new colleagues in person, and putting human faces to inhuman avatars in chat rooms.

And it is precisely because these things are important that we need more than hoping they will happen when we put people physically next to one another.

We need purpose. We need intent. We need new forms of collaboration — and management — adapted to the reality of in-person collocation being an expensive arrangement.

Office collocation is a near-luxury item that should not be avoided altogether but also not be used in the same ways, or as frequently as we did ten, twenty, or thirty years ago.

Moving forward means isolating the good things we expect from RTO and adapting them to the realities of evolving technology and shifting demographics across a society coming to terms with life-altering events.

The buildings may be familiar, but the pandemic was a difficult time of extreme adaptation and, in many cases, isolation.

For now, returning to the office and seeing more faces offers a welcome respite to the latter and an opportunity to reexamine the former.

With concerted effort, we may bump into the best blend of pros and cons between the opposite ends of the morning commute.