Linux distros

Hello and welcome to Ship It, the podcast all about what happens after you git push. I’m your host, Justin Garrison, and with me as always is Autumn Nash. She’s not here right now… We’ve been doing these shorter intros just to kind of intro the person we’re interviewing… I think we’re going to drop these in a little bit, and we’re just going to do proper intros with everyone in future shows… But we haven’t been doing that, and we’ve already been recording some of these shows… So I’m just going to introduce it. Autumn’s here on the interview, and we also have Jorge Castro. And Jorge created a project called Universal Blue, which is a foundation for a bunch of Linux distros based on Fedora Silverblue. We go into it in depth. If you don’t know what any of what that is, Jorge is very careful to call Bluefin and Bazzite not distros, because they’re flavors of Fedora Silverblue… They’re basically just more opinionated Linux distros.

I have been using Bluefin on my developer desktop for over a year now. I have been using Bazzite on my Steam Deck for probably the same amount of time… I love them both. I think they’re great projects. They have Chrome OS-ified Linux distro for me, and that’s been wonderful. So if you want to learn more about it, stick around for the interview with Jorge, and check out the links in the show notes.

[00:05:13.12]

Hello and welcome to the show, Jorge Castro. Jorge is a part of developer relations with the CNCF or the Linux Foundation, one and the same, for some parts… Jorge – but that’s not what we’re going to talk about. Today we’re talking about your side project, this thing that you started and now it’s taken over a lot of things… And I love it, and I’ve been using it for a year. Why don’t you tell us about UBlue?

Yeah, and we are going to bring up the cloud-native thing at the end, because that’s the mission part at the end. But let’s talk about the nerds stuff first. Yeah, so me and some friends started Universal Blue, which is an umbrella organization that publishes Bazzite and Project Bluefin, which are operating systems for your computer. Bazzite is for gamers, for handhelds, and all the fun stuff. The new Asus ROG Ally you can run on your Steam Deck… You can run it on your CPC, and it’s got this cool big picture mode style thing… And then Project Bluefin was my passion project, which is originally what I started with, which is I wanted to make the world’s greatest Linux workstation for people us. And distros weren’t really filling that need, and then at some point Fedora gained the capability to map it all with cloud-native tools. That made it really easy for me, and that helped me get people you, Justin, to help me finish it out and kind of build what we want.

Now we are on year three, we’re about to make a release, again, this fall… And things are going great. We’re about to hit 20 million pulls on our images, so I’m pretty happy about that… And yeah, that’s what brings me here.

That’s awesome. And I remember the first time you told me about it, you told me the domain first. It’s the letter U, blue the color, and .it. It was the very first domain that you’re ublue.it. And I was “Yeah, you’re going to blow it if you create a new distro.” What are you doing? Why are you doing this?

Yeah, well, for getting domains is hard… And at the time it’s “Will this even work? I don’t know…” So you point at laptop and you say “You blew that one. You blew it!”, pointing at your ThinkPad. So it was actually intended to be a slug, and then one day when we actually realized that the model would work - because you had to figure out if the model is going to work in the first place before making a product. And once the model started to work, it became more clear that it wasn’t going to be a weekend project anymore. It originally was a weekend project, and now we’re on year three.

Describe the model. Describe what you mean by “You have to make that side of it.”

Yeah. So the model basically is – Red Hat and Fedora had this thing called RPM OS3, which I don’t really want to get into… It’s basically just an operating system in a Git-like structure. So imagine that your entire operating system is just in Git.

[00:08:01.12] I always describe people go to the root of your file system, do git init, and git [unintelligible 00:08:02.07] and that is basically –

Exactly.

And then you go check out that in another folder, and you say “Oh, this one’s main, and that one’s for–” whatever branch you’re on. That’s basically OS3 in a nutshell.

So what’s cool about that is the entire OS in there - the kernel all that stuff. So it was around for a while… People use it in embedded systems, and stuff. And it was neat, but it was decidedly niche. There’s always that bit of embedded tech where you’re “Well, that’s pretty cool. I can’t really use that anywhere else.” And then at some point at KubeCon I ran into Colin, the author, Colin Walters, at a KubeCon… And he goes “Hey, what do you think about this? I made it so you could use all that stuff, but you can model it inside a Docker file.” So your Docker – you just make a Docker file, and then you say “from Fedora”, and then you do RPM OS 3 install your packages, and then you add a few extra commands - there’s a few extra differences - and an operating system came out. I was “Really? That’s weird.” So I did that, I added a different text editor, and then I made a copy statement you would in a normal container, except I copied a wallpaper over… And I put that in GitHub, in a GitHub Action, and some friends helped me, and an operating system container came out.

It was eight gigs, because it has a kernel, it has the full system… And I built it in GitHub, and then GitHub shoves it into the GitHub registry. And then my laptop pulls from it as if it was pulling from your app repository. So it gives – now anyone who can make a Docker file can customize operating systems. And at first – usually we’d talk to server people, “Oh man, I know exactly what I’m going to make.” Right? So I don’t think that was the tough sell, but I always kind of to do things the hard way; it’s “Hey, wait a minute… I want to make a Linux desktop with this.” Because Linux desktops are still kind of traditionally package-based. They never really went through their cloud-native journey… So when you’re used to working with cloud-native tech and you go back to the Linux distro, it’s literally going back to the ‘90s, it feels like, sometimes. But there’s a lot of good stuff there…

So this allowed us to basically take a lot of the good stuff that we love about Linux distros, but shove it into a cloud-native model, that now I can use all the tools that I have in GitHub… So I spent about a decade at Canonical, building Ubuntu and Ubuntu Server. And the amount of infrastructure it took back then to make an operating system was immense. You had to build Launchpad… Which is – before GitHub, everyone was trying to make one of these. And the infrastructure it takes to make an operating system and all that stuff is really hard. But now, because Fedora just made it that it’s just a Docker file, I can use the same tools and stuff that people just used to make their Node app containers, except I can put an operating system in there.

So that’s what we did. We started publishing base images, and then people Kyle came along and said “It’d be really cool if I were to stick gaming stuff in there”, because trying to get gaming on Linux is hard. There’s multiple configurations, tons of distros… It’s just – you’ve all used Linux beforel; it’s just a hot mess, you know?

So the pattern really enables what we were able to do at work with cloud-native, which is why I still kind of mentioned the cloud-native stuff… It lets distribution people who don’t want to deal with RPMs, and all this low-level stuff. You let Fedora and the distros handle that. And then we handle the customization bits, the kind of SRE style things, in the same way that your developers develop the app, and whatnot. And then at some point that gets git pushed, and then there’s a container, and then there’s an ops team that you work together with to get that to production.

Those are the patterns that I wanted to bring to the Linux desktop. Because the value isn’t actually in the artifact. It’s the fact that you can gather experts around a Git repo, and apply that to your workstation so that when I fire up my VS Code, I just want my VS Code to be the good one.

[00:12:01.17] Brian Ketelsen, who’s on the team, when we were hanging out, I was like “Show me your VS Code setup.” Did you ever get to know someone, and you’re like “Show me your VS Code setups”?

Once you know them enough, you’re like “Hey, what’s your vimrc?”

Yeah, exactly. You’re doing the nerd stuff. And then you’re like “Wow, let me nick this. Let me nick that. Let me put this in my toolkit. Let me introduce some tools to you.” Atuin is one of my favorite tools, which is a shell history thing. And we just started to collect all these little nerd tools or whatever. And it’s like “Well, why don’t distros ship this out of the box?” Well, it’s an opinion. So if we can make a thing – I made a thing that’s called Project Bluefin. This is going to be Linux for the rest of us, and we’re going to have the cool Linux tools. Brian’s VS Code config is basically what we ship, with the dev containers and stuff already installed, ready to go… So we wanted to kind of prove that the same level of efficiency and stuff that you’ve got from cloud, you can bring to your workstation, and then just have it set up out of the box.

See, I love that so much, because I have a Mac, and I run Linux on a dev desktop, but I use my Mac for everything else, because I don’t want to deal with – how you said, it’s like going back to the ‘90s sometimes. So I’m just like, I could run Linux all the time if I use Bluefin; that’d be cool.

And how much of that is – you’re shipping opinions, right? One of the best things that people like is “Oh, look at all this choice we have on Linux.” Linux desktop… I can pick anything I want. And in this case, you’re like “Well, no, we’re just going to give you what we think works well”, especially in this sort of like - I don’t wanna say cloud-native space, but… If you’re familiar with containers and you’re doing that sort of stuff, this might fit your model better than something like Nix. Because Nix has its own set of opinions and how to manage things… And if you’re doing Docker files on top of Nix, you have two layers of – this is more opinions, and more complex sometimes. You have to kind of switch your model around how things work.

Yeah. I grew up on traditional Linux. I grew up on Enlightenment and Window Maker, and stuff.

I mean, I loved Enlightenment.

And in the early days when you’re very first getting started, you’re very much focused on “I’m going to make the perfect thing for me. It’s going to have –”

I was getting my wifi drivers. That’s what I was focused on.

Well, also that. Okay, so assuming – so after the first two years of getting the right hardware…

XConfig and wifi drivers, that’s it.

So when we are growing up though, it is all about finding your perfect setup. And we grow up seeing all these Linux screenshots, and it’s all about the individualism.

It’s like Compiz, Beryl, that stuff…

You want to express yourself. And Linux is – all the other kids have their traditional desktops… Your coworker has his golden image from Work IT, or whatever, and Linux is like your hot rod. It becomes part of your identity. But as we got older and the open source movement gets – new people are coming in, people are cycling out. Now that the real long-term sustainability is our issue, at some point you’re like “Okay, who’s got the best one of these? Brian? Okay, we’re going to start with you, and then we’re going to make parameters for all of the things that we want an opinion on, that we feel is important, but let’s go into it saying that 90% of the stuff we’re just going to manage out of our lives.”

So for us with Bluefin - you like Gnome, but you want Ubuntu-style Gnome; sort of the 1990s X-Men. I like to think of old Ubuntu as the ’90s X-Men…

I love that.

That’s my X-Men. That’s the X-Men I identify with the most. It’s the same way with technology. Because it’s the same, but it’s also different, like what multiverse of your favorite superheroes… You have that with Star Wars, what era is your favorite, sports teams or whatever… So if you grew up in this kind of classic Ubuntu era, [unintelligible 00:15:56.00] that’s kind of awesome for you.

[00:16:00.13] But you’re looking for something more modern, something more standard space, something that really – what I try to do, especially with Bluefin DX, is set people up for success. Pretend, when they open that laptop and there’s Bluefin on it, that they’re at a workshop, learning their first deployment of something.

I was just going to say that it makes it more accessible. And then you can learn to change all of those different opinions as you go on. If you took, I don’t know, different students that we teach at Milspouse Coders, or different kids - there’s some people who really want to be in those weeds, and they really want to change everything and learn every little piece. And there’s some people where they’re going to get frustrated and not want to even try if it’s so down in the weeds. So it’s that’s really cool, that you’re giving people the options and making it more accessible for maybe somebody who wants to get started.

Also, sometimes – at work, I want all the different opinions, and stuff. But when you’re at home and you’re using your laptop and you want to do something fun - dude, I just want it to work, and then I want to change what I want to change.

Well, and as a user of both these products – I installed Bazzite on my Steam Deck more than a year ago now. And I haven’t changed anything. I just keep doing updates… And my son runs it on his Steam Deck, and it’s his main computer; that’s his school computer, but he also games on it. It’s just “Yeah, it’s just Bazzite.” Because you get all the flatpak stuff, you get the things that Steam didn’t really integrate well into SteamOS, because the desktop environment was definitely an afterthought. It was like “Yeah, you could also do that other stuff if you wanted to. But you probably don’t want to.”

And then on my desktop, with Bluefin, when you told me there was an NVIDIA build of it, where it’s just like “Oh yeah, I just want the developer NVIDIA”, I was just like – I just rebased off of it, basically. It was like a rebase reboot, and I’m like “Wait, I have NVIDIA now? Hold on a second. I didn’t go through any install process, I didn’t have to do all this–”

It’s like building the module locally, on your computer… It just doesn’t make sense. It’s very interesting to me, because if you were to come out with a traditional Linux desktop today, it’d be the worst product, and people will call you out on it. But it’s been around for so long.

So the idea that you have – let’s say your Linux distribution has a hundred thousand NVIDIA users, and you’re expecting 100,000 people, computers of different manufacturers, to all compile a kernel module, to load a binary driver, and you have 100 shots at it. What are the chances you’re going to get 100 out of 100? Keeping in mind that sometimes you don’t even know what’s happening, and you might turn off your computer and it absolutely fails.

Whereas the cloud-native model is like “Well, let’s build it one place in CI, and if that passes, it passes. If it breaks–”

Don’t promote.

Yeah, don’t promote. Exactly.

What is that infrastructure? Where are all the builds taking place? How are you distributing this? This is a Linux distro, and like you said, distributing Ubuntu took a lot of services and products, and now you have a modern Linux distro. What are you doing with it?

Everything’s in GitHub. You can run your own personal image out of GitHub as well. We pay, because we want more concurrent builders, so it costs about $70 to run the entire project, a month. And we take in about $80 in donations. So for me, that’s the perfect – you know, just enough budget over in case there’s an emergency.

So we actually went into it with a “We don’t want to make infrastructure.” Because one of the things I did learn about making a Linux distribution is it’s hard, and everyone falls into the trap. “Oh, we’re going to need our own web server. We’re going to need our own mirrors. We’re going to need our own this. We’re going to need our own that.” Where our mentality was “Let’s leverage as much SaaS as we can to get what we need done.” And then our ISOs are served by CloudFlare, which is R2, and the cost is fantastic.

0 dollar egress. Yup.

[00:19:55.20] I think we pay $8 a month for all of Bluefin’s ISOs. It’s such a great deal. And a lot of these companies, they have fantastic open source programs that we take advantage of, which – we try to do our part, too; recognize when companies are supporting open source. So Discourse, that runs our forum, has just a great open source program for communities that meet a certain size. You don’t just get it right away. You’ve got to earn it. And that really helped us. It kind of gave us – when we reach a certain size, we know that we’ll be able to leverage these kinds of things.

So we really do try to run it as much as possible with SaaS, and things like that. That helps us keep it lean, because it does take a lot of effort to run a Linux distro, and we want to leverage as much goodness from Fedora, but we want to give our users what they want, in the most efficient way possible. We could definitely set up a lot of infrastructure. We say no to ourselves all the time.

Equinix gave us a really nice ARM builder. It’s just awesome. But trying to get bare metal up and running for your own build… At some point we’re like “Okay, we’re just going to wait for GitHub runners.”

Yeah. I feel like that’s working smarter, not harder sometimes, because sometimes you get so into the infrastructure and doing everything yourself that it takes away from building the main product that you want to build. And you can always go build the infrastructure later if you wanted to.

Well, and by keeping all of this SaaS-based, what’s the process for me to go make my own fork or build of Bazzite? You can just do that, because I don’t need to have a dedicated runner. Like, I’m still using GitHub.

Yeah. So in the old days you’d have to fork all of Bazzite, and then keeping up with that thing. Not awesome. These days we have an image template that we made, that literally says “from”, and then we commented out all the sections, and then you would do Bazzite.

Oh, that’s smart.

And then you would add the packages that you want. Removing packages is kind of weird, because it is a layer…

Because it’s Git?

No, because it’s an OCI container. So if I give you Bazzite –

Oh, popping the – yeah, yeah.

Yeah. Because that’s a layer up. You can’t remove things. However, we have this fella Amtheus, who’s been figuring out a re-chunker that will go back and reorganize all the layers and remove all the stuff that you’re not using, to get the image size down. So the image size and the updates are still not optimized, but Red Hat is productizing this for their enterprise stuff, so as they figure that out, we get the advantages. Because their [unintelligible 00:22:18.08] containers are like 300 gigs, or something. And we think Bazzite is bloated because it’s 20. [laughs]

Again, by using things OCI containers we could just leverage existing tools, optimization tools. On your OS image, you could run Gripe, the tool that you use to find the CVE vulnerabilities in your app containers… You can run that on your OS container and it’s smart enough to give you all of the versions of the packages, and stuff. They did a really good job normalizing the OS and what it would look in a container, in my opinion, which leads to our success.

It’s really cool. Just the way that you took all these different building blocks and used them together… It’s my favorite part of programming. It’s like building Legos, but with computers.

Yeah.

And this is the first distro that I’ve ever seen that encouraged you to go make a fork. Because the fork is not maintenance for me. It’s not like you have to go maintain this whole thing. No, I can just keep pulling upstream, rebuild it, and I can rebase off of my own published image, with the 10 or 15 packages that I want… And then I pull them in and I’m staying one level behind what upstream is.

Yeah. And really the only maintenance is - you’ll find this out - GitHub, if you don’t make a Git commit to your repository and you’re running Actions, if you don’t make a commit, it assumes that you abandoned it, and gives you a warning “We’re going to turn your Action off”, which would actually turn off your OS. But yeah, literally, other than checking in every once in a while to see how your build is doing, it is very low maintenance.

[00:23:55.03] Although what we are finding is initially, I think, because people were also learning how to use a Linux in this manner, instead of their traditional way, I think a lot of people go into it thinking that they need to make a custom image, and a lot of people do follow that pattern. But what we did find is a lot of people do end up, even after you do your own image, being like “Wow, I’d rather have someone else do this anyway.” And as we bring in new tools, it alleviates a lot of the need to do a custom image. So getting Homebrew, I think, in there was a huge win. I know a lot of Linux traditionalists don’t like Homebrew, but –

They can stay in their old-school – I’ve been using Homebrew for four or five years on Linux, and it’s fantastic.

Yeah. And then for us, because we’re – obviously, with Bluefin especially, I’m going after Mac people. Especially our open source people.

You are going to convert this Mac person, because I’m sitting here like “This sounds amazing.” For instance, some people really get into that wanting to do all the really lower level stuff. I was a kid who wanted to customize things, but at a different level. So you’re speaking to – I feel like you’re speaking to so many different people, and making it such a big audience, because now you get to choose the level of abstraction of getting to pick those things.

So to me this sounds great for development and for personal, because I get to pick where I’m – if I just want to do something and I want it to work, it does that. When I want to customize it, it does that, and it gives me all the pluses of Linux.

Right. Or your brew file.

So you’re converting me as a user.

You’re right, your brew file just works. You just install it.

Yeah. The GUIs don’t. One of the things I want to do - if there’s any Rails nerds or Ruby nerds out there, is I would love to have a cask that lets you use a brew file. And you know, on a Mac it installs the Mac GUI applications… I would love a cask for Linux, that would let me define my flatpak applications in a brew file, so that I can just brew install Firefox and it will get the right flatpak.

I need a little bit of duct tape there, if anyone’s interested. These are the kinds of little things that you’re going to find, where you’re like “Oh yeah, brew and Linux sounds amazing.” But then you’re also missing a few things.

But that’s not missing something that would be a huge blocker, because I think a lot of people want to get into Linux, but sometimes it can be a little intimidating.

And there’s too much choice. All of the benefits of all that choice is paralyzing.

Yes. Sometimes you just want to start, you know… And I think this is a great way to bridge that gap and to give people the start. But still, you can learn, and you can go and look and experiment, but it’s enough to get you started.

Break: [00:26:39.23]

Is your sticker the one with the little raptor on it?

The little dinosaur. Yeah.

Oh, my God. My kids are obsessed. Those stickers are all over my house. Justin gave them stickers, and they’re like –

Oh, I’ll bring you some. We’re getting new dinosaurs, and stuff. I work with paleoartists, because that’s the whole dinosaur evolution thing, and…

We’re about to be your biggest customer, because I want to try your product, and my kids are obsessed. They’re all over their little laptops. They got their first Chromebooks this year, and it’s all over their iPads, and they love your stickers.

That is so validating. Thank you. I can’t wait to [unintelligible 00:29:52.26]

I just want you to know that they’re little dinosauroholics and tech nerds, and they love your stuff.

Yeah. I’ve got a whole species…

Oh, that’s a new one. That’s so good.

Yeah. This was called Markovinator. This one is –

This is an audio show, so we can’t –

Oh, I’m sorry. So I got one called Carl… This is on the website. This is the big, towering one.

Do you have an Instagram, or anything?

No, but you can buy them directly from the artist. I’ll make sure I send you all links, and things like that.

Okay. If you guys ever make shirts, I just want you to know I have tiny models, and I will buy the shirts, because each of them got for their birthday their own little Chromebooks, and you know how my computer laptop has so many stickers, so then as soon as we got back from Scale, they started putting all the GitHub stickers and all the Bluefin stickers… So they’re so into it. They think they’re baby devs now. It’s adorable.

Not to do any spoilers, but I do have a plan bringing my kid to Scale, and then there’s little dinosaurs, and stuff, and they’ve got their –

My son’s going to be there next year.

I’m bringing my kids, too.

Oh, okay.

We’re rolling up with a gang of kids, because I’m bringing my kids, Carrie’s bringing her kids… We’re trying to get the two college students to give talks…

Scale is the best place for families. It’s the best conference for –

I love Scale.

And their dad lives in California, so I’m going to be like “Okay, come for one day”, and then I’m going to do fun nerd stuff with my friends.

Yeah, that sounds fun.

I’m so excited.

FYI, just a side, too - anyone that’s interested in Scale, the CFP opened for Scale last week, or something like that. So if you want to submit a talk, it’s a great first time place to speak, because it’s a small community conference, there’s not a lot of pressure…

It is the most welcoming, kind… You can ask questions, nobody acts like you’re stupid for asking questions… It’s like nerd spring break over there. I have never thoroughly enjoyed a conference – it’s not fancy, but it is all the good parts of just like…

Yeah. And it’s affordable.

Yes.

It’s a community thing.

One hundred percent.

It’s a hundred bucks, or something like that. It’s in Pasadena in March…

But it’s accessible, too. Do you know how exciting it is to be a mom who’s a software engineer? People are always like “You can’t be a mom and a software engineer.” I walked up, there were babies in talks. People brought their kids; you get to bring your whole self to Scale. You know what I mean? And I love that. Also, if you guys need any help, I’ll totally jump on helping this year, too. I’m about to submit my talk. I’m going to submit my Barbie CI/CD talk to Scale this year.

That would be amazing. Yeah, the team’s going to try to go to Scale. A bunch of them went last year, but I wasn’t able to go. So this year I’m fighting to go.