Optimising sociotechnical systems

Ben, welcome back to Ship It!

Great to be back again. I can’t believe it’s been a year since our first round.

Do you remember the episode, the one when we first recorded? The episode number?

Yeah, it was – well, this time last year; when we chatted again recently, I couldn’t believe it had been a year. But I check back on my calendar, sure enough, a year ago. Crazy.

So that was episode four. If it was five and this would have been 55, that would have been perfect. However, by the time you’re listening to this, it will be exactly one year, give or take a day, from when the previous one came out. That was episode four, “OODA for operational excellence.” What is new with you?

Well, still definitely going around that OODA loop. Since we chatted, I’ve been doing a lot more research and exploration of the OODA loop. I was very privileged to be part of a five-series conversation which was streamed on YouTube with some OODA experts, and we had some really, really interesting discoveries… Not really discoveries, but insights into OODA and things like that.

Over the last couple of years I’ve been trying to figure out how to build better OODA loops for businesses through technology. By the time this goes out, I feel fairly safe to say without touching wood and making a tap in the microphone that that company will be live and will be rolling that out. So… Very exciting.

So OODA still works. That’s the important one. OODA as a principle works well.

Absolutely. I do a sporadic podcast with a friend of mine and one of the things we say on there is OODA is pretty much axiomatic at this point. Whether you call it OODA or whether you call it something else, the thing that’s going on there is going on, whether you like it or not. You don’t do OODA, OODA does you. [laughs]

Wow, okay. Why do you say that? Why do you say that OODA does you? That’s a very interesting expression. I haven’t heard it before.

Yeah, it’s because people always say – you know, there’s this move within our industry… There’s this over-focus on frameworks, and we’re going to use this framework, and we’re going to do this methodology, or we’re going to… Pick your poison there. But OODA is not something you do. OODA is something that happens to you, because you are an entity moving around within your environment and it’s a biological imperative that you improve your capacity for independent action, whether you as an individual person or a business.

So OODA is happening whether you like it or not, and whether you call it OODA or not, because there’s plenty of other things that approximate Boyd’s work out there. So at this point, all of the exploration and research I’ve done, I do believe that it’s just something that’s happening, and whatever labels we use doesn’t really matter, because underneath there’s that principle of reacting to the environment that literally everybody has to participate in.

So a lot of great words were mentioned in these few minutes, and I’m going to mention another one, which I think goes really well with this… And then you have to tell me what generator do you use the generate your talk names, because I think they’re spot-on. Maybe it’s you… [laughs] You gave a talk, The Paradox of Control, and I really liked that title. I really like the way you formulated it. So what gave you the idea or how did you come up with that title? I thought it was an excellent one, by the way.

So this was a talk that I gave at the NATO Command and Control Center for Excellence last November. The C2COE is a kind of a practitioner body that is aimed at improving military command and control systems. My background is – when I was in the military, when I was in the Royal Marines, I was a signaler, so I was one of the low-level cogs in the command and control system, if you like.

So I went and looked at all of the recent research, breakthroughs, technology, approach, and I put that up against what I know of OODA and what I know of evolution and ecosystems. What I’ve found was a paradox - it’s that the command and control is over-focusing on the control part, and it’s focusing wrongly on the control part… Because the more you try and control a complex system, the less in control it is. Example - forest fires. With the huge forest fires that are happening in the States, there’s quite a strong argument there that one of the reasons they’re so huge and so out of control is they’ve been trying to control the outbreak of these fires for so long that it’s built up loads of potential in the system, and now they’re completely chaotic.

So I feel like can command and control, when put into the context of the military procurement system, you end up with these big programs worth millions or billions of pounds or dollars, very technology-first, very much designing a system that they expect people to jump through the hoops of… And I experienced this in the military, being given a piece of kit that was utter piece of rubbish, and being expected to use it, having never been – apparently no one on our level ever being consulted about the development of it.

Can you tell us what was that piece of kit?

So back when I was in the Marines - and this is going to make me sound extremely old, but it just goes to show how out of date the military system is…

Experienced. Seasoned.

Experienced, let’s go with that. That’s fine, I’ll take it. We were still using analog radios, believe it or not, around about 20 years ago, despite the fact that mobile phone networks were a thing, satellite communications were a thing, 3G was a thing, digital comms was a thing… But we were still using these God-awful radios that were developed in the ’60s and ’70s. Crazy.

Reliable, maybe? Was that the reason?

No.

No.

I’ll tell you a story in a second about probably the reason I left the Marines actually, but…

It wasn’t because of the radio, was it?

It was. It bloody was.

Really? Oh, my goodness me.

I’ll tell you in a second. Let me just finish this thought and then I’ll go back to that. So the piece of kit that we were testing was a way of securely communicating over HF radio. And HF radio, it’s a ridiculously low board limit because the frequencies are very low. You can’t physically fit much signal into it. And it was just awful. This was in the days where you used to encrypt radio signals by pulling a punched piece of paper through a physical piece of equipment. If you got caught by the enemy, the first job you had to do was burn the piece of paper so they couldn’t decrypt your signals. Seriously. 20 years ago, this is.

Wow, okay. Are you sure it wasn’t 50 years ago? Ben, just how experienced are you?

No. It was developed 50 years ago, but not updated for 30 years. So anyway, the story of the why I ultimately decided to punch out of the Marines and think that I had better things to do with my life… 2003, just after I’d been teaching myself to code, I was sat in Saudi, and this was just before we did the assault on the al-Faw peninsula as part of the Iraq invasion.

My job was to get all the comms ready. Bearing in mind, these communication systems are radios that you set up. Back in those days, you used to hand-tune the antenna lengths to make it match the frequency of what you were talking. We were trying to get comms with a ship that was maybe 50 kilometers away.

Right. Wow.

So there’s me and all of my buddies, sat in the desert for three days, twiddling antennas and moving masts around so they don’t interfere with each other. It was an absolute ball-ache. Then the Navy SEALs rocked up, because they were on the assault with us. They were doing a practice session with us. And they rocked up probably a few hours before kickoff time. Fairly nonchalant.

This signaler, my counterpart in the SEALs, pulled his truck up, turned it around, reversed it up next to our command post tent, got out of his truck, pressed the button, up went a hydraulic mast, and he picked up his handset and he had comms immediately. So what had taken me three days took him approximately three seconds. And I already knew that there was a better kit out there and…

So that was it. That was the moment that broke you.

Yeah, that was it. “The mission doesn’t make sense to me.” So that’s one of the reasons I’m now doing what I’m doing rather than being a sergeant major in the Marines or whatever. [laughs]

So compared to that, Kubernetes must be child’s play. Still, you don’t do that, because complexity - there’s better ways, and there’s a different type of complexity that you want to spend your time with. How does this compare to the technological landscape and the infrastructure landscape that we have today? Because I see a lot of similarities, but I’m wondering which ones are you seeing.

That is a very, very astute observation, because the dynamics that played out, of our kit being behind the times - you can see that playing out in reports about the Russian kit now. But the dynamic is exactly the same as what’s playing out in the technology environment. It’s obsolescence, and it’s an evolutionary struggle of ongoing improvement. Can you imagine if you had a piece of technology now that was 50 years old? We’re 20 years in front of when I just had that story.

No, I can’t… No way.

You just cannot stay competitive with these long procurement cycles. For us in technology, that means that the time horizon or the half-life of your current systems gets shorter and shorter and shorter, so you have to be continually reinventing yourself. If you launched a tech company five years ago and you became successful, and you’ve now got competitors… Say you launched five years ago and you launched on, I don’t know, AWS, maybe you were using Kubernetes at that point, maybe you’re using Docker, but probably you were using virtual machines, maybe you’ve terraformed it up, maybe… But you’ve become successful, you’ve gained a load of internal inertia, and now you’ve got a competitor snapping at your heels, and that competitor is using something like Vercel, using something like Fly.io, or railway.app for their infrastructure. They will be building - it might not be an order of magnitude faster than you, but it will be faster than you. Exactly the same dynamic of what I just shared about the radios back then is now in effect at probably two, three, ten times the rate in our daily lives as technology professionals.

Okay. I had a recent conversation about this, where we were talking about Terraform and ECS versus something more modern, like Hasura or Fly.io. People keep reverting back to what they know. So there’s an element of being comfortable, and they say, “No, I’ll be quicker, because what I know will make me quicker.” What would you say in those conversations? What would be the argument that you would use for the new wave of infrastructure, the new way of tech?

Yeah, I’ve come across this argument, and it does make sense in certain contexts, I think. Government IT is a good example. In the UK, in many departments in the government there’s this mantra of “Use boring technology.” There’s even a website for it; I can’t even remember what it is, BoringTechManifesto or something. Again, this is something that may have made sense five years ago.

If you’re five or ten years ago and you had the choice of all those sexy technology or stick with Java, then that might be defensible back then. But if you had the choice now of choose something that is a huge CI/CD pipeline, multiple days to get anything out the door… Technology moves on, and you can always make that choice of familiarity giving you that immediate getting-off-the-ground advantage, but it’s always a trade-off. You will always have to be knowingly trading off that there are other teams around that are using technology that’s more modern, and they might not be even as good as you… Good if there is some sort of objective measure of goodness within a technology team; but they will be using better tools.

Like with anything, there’s no right answer. But if you’re a company that’s maybe fallen behind the times a bit, there is a very strong argument for saying, “If you’re going to make a leap forward, make a leap forward right to the cutting edge of this evolutionary landscape, rather than making an incremental step forward.”

Okay. I’m wondering how much of this is linked to that – you know, when you create and destroy, create and destroy. Some call it disruption. I think you have another way of phrasing it. How much of this plays into that? …in that, if you’re comfortable and if you say, “No, no, let’s just do what always worked,” what got you here will not get you there, and you’re trying to get there. You’re not trying to get where you’re at. How much of that do you think translates into our technological choices? Whether it’s Kubernetes, whether it’s VMs, whatever we choose.

Yeah, that’s a great question. There’s always the pressure to get started as quick as you can. That’s the whole lean startup, all that kind of thing. So given that you start a journey with the resources at your disposal, there’s obviously a strong incentive to begin that journey with what you have right now. Take a small step forward, prove that it works.

I was in a really great conversation last night, actually, and the issue is that it’s not a turn-based game. People don’t wait nicely in line, your competitors don’t wait nicely in line, the market doesn’t wait quite nicely in line. It’s not chess where you take a move, somebody else takes a move; everyone’s moving all the time. So the problem with comfort is that you can become accustomed to being comfortable and you’re not ready to be uncomfortable. The more comfortable you stay for the longer time, the more uncomfortable you’re going to be very very quickly when the situation changes, because you’re just not used to doing new stuff.

You can see that in – government services, and again, I’ve mentioned them before… But the pressures that society faces as a result of technological change are now butting up against - across all of the world really, apart from maybe Singapore and some other forward-thinking places - are now butting up against the inability of civil service, political systems, to change. And when you’re now looking down the throat of it, economic crisis and climate crisis…

I was listening to the radio on the way back from dropping my little girl off to school and they were saying that, “We’ll see, we’re going to get planning permission in two weeks or a couple of weeks.” I’ve worked in the Planning Inspectorate a little bit… So that’s probably the end two weeks of probably several years of back and forth and utter waste of time. And now we’re going to begin building, hopefully in the next few months, and we’ll have a new power station in 10 years. Wow.

[laughs] Wow. Yeah. I think this is so relevant. The energy sector is going through some huge change. We’ve been talking about renewable energy for a long, long time, and there are certain events happening around the world which have accelerated this. But is it… I don’t want to say too late, because it’s never too late… But is it maybe five years too late? We could have done this five years earlier; why do we have to wait? But then I think there’s something in us… We are waiting until we can wait no more and then we make the change. Maybe there was a better way. Maybe, I don’t know.