Lessons from 10k hours of programming

Matt, welcome to the Changelog. 10,000 hours is a lot to put into anything, and at some point you hit mastery… In your blog post on the subject, titled “Reflections on 10,000 hours of programming” you quoted Malcolm Gladwell from Outliers, “The key to achieving world-class expertise in any skill is to a large extent a matter of practicing the correct way for a total of around 10,000.” So 10,000 hours to master a skill, that’s where we’re at. You’ve got some lessons here you’ve shared, reflections for you, but lessons for us, so let’s dig into those… Where do you begin when you reflect on 10,000 hours of anything?

Well, just when I think about 10,000 hours - I mean, it’s a long time. I think about how long I’ve been doing this… I’ve been programming for probably 15 years now, and it’s a lot of time to do anything. I’ve had tons of failures along the way, learned a ton of things, and I’ve been trying to blog more and write down these ideas so that I don’t keep on making the same mistakes over and over again.

Ha-ha…

So it’s a lot for me as well.

It’s the DRY principle that I do adhere to, Don’t Repeat Yourself when they’re mistakes; don’t repeat your mistakes. DRY them, as they say.

Oh, yeah.

What is 10,000 hours? So if we assume eight hours a day, five days a week… Let’s just say eight hours a day times five, right? Times - call it 50 weeks a year… That’s 2,000. So if you’re working a typical 9 to 5, take a couple weeks off for vacation, that’s 2,000 hours a year. And you’ve got 15 years, so you’re well over the high watermark. Did you do the math, or you’re just like “Yeah… I’m there.”

No, I did the math. And I’ve spent a lot of time in open source as well, so it’s not even a 9-to-5, it’s like a 6-to-12, or whatever. I mean, it’s an all-day thing.

[04:07] So you’re well over. Where did you earn your keep? You’ve had a couple different jobs… Do you wanna tell us about the 10,000 hours you’ve put in? …where it was and what kind of stuff you worked on.

Yeah, so I’ve just been programming a bunch… I programmed a bunch in school, after college I worked in New York for a bit as a programmer… I came out to the West Coast here to work at Google. I worked on open source, I worked on Kubernetes, and specifically a bunch of sub-projects in Kubernetes. I was a maintainer of minikube, kind of the local development environment for Kubernetes. Scaffold, which is kind of a Kubernetes tool to help you build and deploy your apps… And then Kubeflow, which is a machine learning toolkit on top of Kubernetes as well.

In addition to that, I’ve just been kind of hacking on all sorts of open source projects. I wrote this configuration language Virgo, which is – you can think of it as like if Yaml was for graph-based configuration instead of more hierarchical, and then built a computer vision bot for Runescape, which was just a game that I used to play as a kid a ton…

Nice…

I learned a lot about programming through that, just because, you know, I was too lazy to mine the rocks or click the buttons all day… And just tons of projects like that.

Awesome. Well, you can learn from experience, but you can also life-hack - learn from other people’s experience. So I loved this post… You’ve got 31 things that you’ve reflected, and it is specific to programming. These are not large life lessons or more people lessons, you say; these are specific programming lessons that you’ve learned… And I thought “Let’s get some of these out.” We’re not gonna cover all 31 here; we’ll reference the blog post, of course… But it’s nice to have the one-liner, because it can resonate with you, or maybe shock you, but then I think it’s even nicer to have a conversation around these things; hopefully, they become ever more sticky or more real to people.

So we’re just gonna go down, pick a few, see how long we last, and talk about some of these reflections of yours. Sound good?

It sounds good.

I have a bonus for our listeners too, by the way… Since we don’t know how many we’ll cover, and there’s a free T-shirt in mind here - I’m curious if someone can listen closely, and the first person who can say how many we cover - if we cover all 31 or not, or at least how many we cover - in the comments, gets a free T-shirt. So the first person to do that - comment, get a free tee. What do you think, Jerod?

Sounds good. We’ll have to partially cover a few, so we’ll have arguments over “Was that actually one…?”

That’s right. It could be ambiguous, right?

We won’t use any of the real words… All the words have been changed to protect the innocent. It sounds good; free T-shirt… Why not, right?

That’s right. It could be like The Price is Right - just don’t go over.

Okay, don’t go over. [laughs] Adam will post the official rules in the show notes.

Best effort, get the T-shirt.

This audience is software developer; we are pedantic, so we wanna have the specifics laid out… In code, if possible. Can you put it in a smart contract, Adam? That would be appreciated.

Yeah. I wanna write it in Ether, honestly. It’s gonna be fun.

Alright, let’s pick up on a reflection… This seems to be perhaps your favorite, or you said you wrote some configuration language yourself… Here’s one about configuration. I had not heard of this… This is reflection number 30. Oh, I probably shouldn’t list them now, because we’re making it more difficult. This may or may not be in your list… And this is about the Heptagon of Configuration. Matt, I’m gonna let you explain that to me, because I’ve never heard of this before.

Yeah, you’ve probably never heard of it, because I tried to come up with it myself… I tried to coin the term. So it’s a new thing. But it’s me trying to describe a pattern that I’ve seen in software configuration where a configuration seems to evolve through specific, increasing levels of flexibility and complexity before returning to either hardcoded values or Bash. So you go from hardcoded values, which are the easiest/simplest configuration, but provide very little flexibility.

[08:04] And as the program surface starts to increase, and with that configuration you start to incorporate environment variables, flags, and eventually you wanna start to check that into version control, so you turn it into a configuration file, maybe Yaml, JSON, something like that… And then as you kind of turn on this heptagon of configuration - and I only call that heptagon just because a lot of the ideas came from Kubernetes, and the Kubernetes logo has got the seven points, and it just kind of worked out well…

Right…

But as you’re going from configuration files, you start to need a little bit more extensibility in terms of templating, and I think templating is something that we’re all unfortunately accustomed to a little bit too much… So that’s kind of one wheel on the configuration heptagon, a configuration. And then from templating, you go to kind of a DSL, a domain-specific language, and that allows you to have a little more type safety and a little more domain-specific reusable modules… I’m sure some of us have used Puppet in the DevOps world, or there’s tons of other DSLs out there… But eventually, these DSLs become a little too inflexible. Maybe the requirements change, the domain changes, and then we go back to Bash. So that’s kind of like this never-ending cycle of configuration that I’ve seen. And I saw this a lot in Kubernetes. There was a lot of Bash in Kubernetes, and a lot of configuration…

Yeah.

Maybe we skip the DSL part, and maybe that’s more of kind of a configuration-as-code, or something like Pulumi… But you know, maybe we’ll go back to hardcoded values at some point.

What’s the takeaway there? Just stick with Bash, and everything will be better? Or is this like necessary complexity here? Is this cycle virtuous, or vicious?

That’s a really good question. I don’t know if it’s either. I think it’s just necessary complexity, and I think it’s important to know maybe where you are on the spectrum, because I do think that you need to – you can’t necessarily jump from something like hardcoded variables or environmental variables to going to a DSL. I’ve never really seen that work out… So I think you do need to increase the complexity, but in a way that complexity can be absorbed by the projects or the developers.

I almost feel like the process of iteration is necessary. You almost learn something – you said that as the surface area of the program evolves, it’s almost like this iteration through the flow, this heptagon is necessary to sort of like flush out the brittleness, or the flexibility and the eventual brittleness again of an application… Because you sort of learn something about it. You provide configuration to the user base, so that they can use it in a more flexible manner, and then those flexibilities turn into like “Well, this is now a best practice.” So all those things solidify to now you just wanna hardcode them, so almost everybody uses the same flexible configuration, in some cases. I mean, there’s a thousand different ways you can slice how this is used in the real world, but that seems to be a necessary iteration process.

Yeah, I really like that point. I think it’s a lot about discovering what those best practices are, and starting to codify them in different sorts of ways.

There’s an analogue to this in economics. Benedict Evans talks about the process of bundling and unbundling, and he says in any given industry you’re either in a bundling process or an unbundling process… And it’s cyclical, right? So an example of that is television, where we’re all cable TV, everything was bundled as one, and then we broke out of that individual, on-demand, subscribe to this, that and the other thing, and how we’re like in a rebundling that’s happening; you can see it with YouTube TV, and different aggregators trying to pool together content. And that sounds very inefficient… Your heptagon sounds inefficient, because it’s like, “Well, we’re going around in this circle.”

[11:57] But what I’ve heard pointed out is that progress often looks like a circle when you look at it on its head, like in a two-dimensional plane… But then when you look at it in three-dimensional, it’s more like a helix, where it is moving in a circular way, but it’s getting better as it goes… And I think with software, that’s a lot of what we’re seeing - these iterations, and a lot of times returning to the old idea, but you’re returning to it with new eyes. You’re returning with new tools. And so you are building up, but you’re not like building blocks on top of each other; you’re kind of like circling the wagon, but you’re going up… You know, it’s like a helix rising… Which is slower than we would want it to, but it’s still progress, right?

Yeah, yeah. I think that’s a great point, and I think we’re seeing that play out in the data stack a bit, with a lot of old ideas and tooling around data warehouses… And now that we have cloud data warehouses, you have Snowflake, BigQuery, Redshift etc. we’re bringing back a lot of those old ideas, things like OLAP cubes, there’s analogues to that now… And it seems kind of like more of the same, but it’s really different once you start to look under the surface.

Well, another lesson here is one that we touched on with the Prag Prog fellas themselves around DRY. This is always controversial, DRY… And it’s because we all think about it a little bit differently, or I think that we all misunderstand what their point was. They did point out on that episode when we had their 20th anniversary show that one of the most misunderstood points in the Prag Prog book is the chapter on DRY. So they tried to rewrite it. I haven’t read the rewrite very closely to know if they accomplished clarifying that, but you have a point here… One of your reflections says “Know when to break the rules. For rules like “Don’t repeat yourself” sometimes a little repetition is better than a bit of dependency.” And you link to another blog post of yours called “DRY considered harmful.” Do you wanna unpack that one for us?

Yeah… I mean, the DRY considered harmful - maybe that’s…

Clickbaity? [laughs]

Yeah, a little clickbaity. And you know, I don’t think it’s actually that harmful. I think the way that it’s been dogmatically used is sometimes a little dangerous… But it’s just more of a point about how as programmers we have a bias for abstraction. Understanding that we have that bias and trying to keep it in check, especially when it comes to duplication versus encapsulation… I just think that it’s a path that I’ve gone down too many times, of carving out microservices, or creating service boundaries where there really shouldn’t be… Or prematurely optimizing when requirements aren’t really finalized; and you know, the requirements are never finalized. And just the wrong abstraction at a low level can really cause a lot of issues in terms of refactoring and just added work down the line.

Yeah, I think we fall prey to this because we’re such pattern matchers… And as soon as you spot that pattern, you’re like, “Ooh… Opportunity.” Some of those abstraction layers are the power in software. The ability to build those abstractions are what give us leverage. So every time we see one, we think “Boom! I’m not gonna repeat myself. I’m going to DRY this sucker up.” But like you pointed out, oftentimes that second iteration, that second usage is not actually generalizable, or it looks generalizable until you find the third one, which - you know, just throw another param on the function. It’s what we do. We’re like, “Well, I’ll just throw a true/false at the end of this thing.” Then I have this extra branch in my function because it didn’t actually map onto the use case like I thought it did. So a lot of it is just that enthusiasm of like “Oh, here we go. I’m gonna DRY this sucker up.” It feels so good, but it does come back to bite.

Yeah… I don’t really know how to get around it. I keep on falling prey to it over and over again… But maybe that’s just kind of the name of the game.

[15:48] And what do you think comes out of the falling prey to it again and again? Do you think that it’s a necessary thing that you just learn from and grow from as a result of like just this awareness that it’s not efficient to repeat yourself? Instead of saying “Don’t”, let’s say “Maybe not repeat yourself”, or “Should not” versus “Don’t.” It’s kind of a little softer on the – it’s maybe just being more aware of the times when there are the patterns… As you said, Jerod, the pattern matching - to just be aware that these can lead down bad roads if you repeat yourself too often, and it makes sense to DRY up things…

To treat it more loosely?

It’s like an awareness thing.

Well, it’s worth pointing out what the rule really was, or is, that they point out in the Pragmatic Programmer book… And the repetition is not about code. That’s where we all get it wrong. Anytime you’re repeating code it’s bad, so don’t repeat yourself. So let’s create a function, name it etc. Abstract a function. What they were talking about is knowledge in your system. Every piece of knowledge in your system should live in one place and one place only. But because the acronym was DRY, and it’s such a catchy thing, and it’s easy to remember “Don’t repeat yourself”, as soon as you start repeating something, you just immediately apply it, right?

Yeah.

But that’s not the point. It’s not about the code that you write. Now, some code does represent knowledge, so it does overlap. These things are not completely black and white, but… That was what they were trying to say. Maybe they say it much better in the 20th anniversary edition. But that’s why we all get it wrong. I don’t know, Matt - has anything helped you… I mean, you’re writing this as a reflection, so you’ve obviously thought about it… Do you just tread more softly? I have introduced the rule of three for myself, which I think I got from Jeff Atwood’s Coding Horror blog, where he’s like “You have to use something three times before you’ll generalize it”, because I have found that it’s usually that third use that points out how bad my abstraction is… But I’ve also found out sometimes it’s like the sixth or seventh use, so it doesn’t always help… But it does help me slow down a little bit. Maybe just like, bite the bullet one more time. What have you found?

Yeah, I think the distinction that you made that the knowledge shouldn’t be duplicated and it’s not so much about the code - I think that’s a really good lesson. For me, I try to understand the bias I have for abstraction, and correct against it… So if that means erring on the side of duplication, then that seems to be kind of the most helpful for me, especially on smaller projects where it’s either just me and a few other devs, or just me… Duplication I think is fine, because the knowledge tax is maybe not as high… But on large teams, I think maybe go the extra mile and make sure that you’re not repeating yourself, because the cost of repeating yourself in that context is maybe much higher.

Well said. I had to just practice this discipline yesterday, because I was creating a gameboard for Go Time’s 200th episode; we played Gophers Say, which is their Family Feud edition… And I wanted a visual aid, so I’ve found a guy who had written one on CodePen. I just wanted to like “Just show me that thing” – you know how Family Feud works, right? And you guess it, and they’re like “Show me what the survey says”, and a thing bings, and it shows the number. And I wanted that for the live show, so I grabbed this guy’s CodePen, I just downloaded it… An index file, a CSS file and a JS file. And I started tweaking it, so it would work for ours. And I know I needed seven rounds.

So programmer me was like, “Alright, now I need a templating language”, so I can just template this out and then have a JSON data blob that represents that… And then pragmatic me was like “Dude, just copy and paste this file seven times, and write the actual data into the HTML. You’re never using this again… And if you do, then maybe you can abstract it later. But just repeat yourself.” Even seven times… Because I knew that was it. I was gonna do it seven times, and I was never gonna touch this again. And I had to exercise the discipline, because engineer me had such sweet solutions for how I could generalize this sucker, and maybe turn it into a web app that other people could use. YOu know, that inclination.

What helped was I had to have it done in like an hour and a half, and so I’m like, “Don’t start coding. Just hardcode the values and move on, man…” It’s tough. It’s tough to fight that urge to generalize.

[20:07] Let’s move to the next one… Here we have a reflection of yours on code comments. You say “If you have to write a comment that isn’t a doc string, it should probably be refactored. Every new line of comments increases this probability.” And then you have a link to a more nuanced take, which is from the Linux Kernel documentation, which I did not read, because - who has time for nuance, right?

First of all, tell us what – when you have that and it’s not a doc string, what specifically do you mean by a doc string? And then how did you learn this, and why do you believe this?