Kubernetes Community Day SF Bay Area - Recap

Last week I had the privilege¹ of attending the first-ever Kubernetes Community Days in the San Francisco Bay Area! It was a super cool event, and I had a bunch of great conversations with folks. I loved how it had the feel of a mini-KubeCon, but was way smaller. I was able to attend every single talk! Amazing!

Anyways in this post I’m gonna do a quick recap of my top three favorite talks, in order of appearance. I believe the sessions were recorded and will get posted online at some point, but I don’t see them there yet. I’ll update this post with links once that happens!

You Can’t Handle the Root! Container Isolation for Secure Multi-tenancy by Mariana Moore at Edera

This was the second talk of the day and it was a really great overview of the state of security primitives and isolation in Kubernetes. The tl;dr here is that containers aren’t actually a good security boundary; it’s better to think about them as an abstraction boundary. The reason for this is that containers all share the same underlying kernel, which means that if there’s a kernel exploit, users can break out of containers and learn information about the host or about other containers on the system. And while the Linux kernel is “reasonably” secure, it’s still a big attack surface and it has lots of CVEs. So what other isolation mechanisms are out there for security in a cloud-native world?

Mariana broke down three alternatives: runtime isolation, node isolation, and hardware isolation. Runtime isolation relies on virtualization primitives offered by the hardware, instead of kernel-level isolation. Traditionally this has meant “virtual machines” like VirtualBox or Qemu, but recently there’ve been a bunch of interesting “microVM” projects that try to provide a smaller, more light-weight, “container-like” interface while still relying on hardware virtualization. Firecracker and Cloud Hypervisor are a couple projects in this space, and kata containers provide a Kubernetes interface to microVMs. There’s some really cool stuff happening in this space, and I’ve dabbled with it just a tiny amount, but would love to do more!

The next level of isolation Mariana discussed is “node-level” isolation, where you just run your entire system on a single (bare metal) node, which allows you to directly control everything that’s running and impose whatever security boundaries you want. This is very easy to do, but you lose a lot of benefits of cloud native technology by doing this.

The last level of isolation is hardware isolation. One way of doing this is by running an air-gapped cluster, but another option that Mariana discussed is using cryptographically-secure hardware primitives to provide processes with their own memory space that is provably² inaccessible to any other processes on the system. This is still a fairly new space, and there’s not good support for these kinds of features in Kubernetes yet, but we’re getting there.

The last point Mariana made, which I thought was fascinating and concerning, is that many of these primitives are fairly well understood in the traditional hardware model, but they totally break down when we start looking at GPUs. GPUs (by design) have almost no isolation primitives, and especially with the advent and adoption of DRA in Kubernetes, this is a space that’s rife for exploitation.

Best Practices for GitOps Promotions by Dan Garfield (Argo, CodeFresh, Open GitOps)

Later in the day, there was a great talk about how to actually deploy your software. The tl;dr of this talk is that everybody means something different when they use words like “application”, “service”³, or “production”, and, maybe we should stop overloading all these damn terms and agree on some common terminology.

Dan went through a bunch of “best practices” for developers and organizations who want to “get their change in front of users with as few problems as possible”; he presented from a “GitOps” perspective, which makes sense since ArgoCD is a giant GitOps platform⁴, but I think all of these best practices make sense regardless of your deployment tool of choice. Here are a couple of the insights Dan shared:

First, pick a versioning that actually conveys information. Lots of people like to use SHAs, because it “uniquely” identifies the code that’s running, but these are actually the worst version indicator. They convey zero information about what’s actually running. SemVer is slightly better, because at least it gives you some ordering information (this version comes before that version). But even this isn’t perfect, because Docker tags (which commonly reference a version number) are mutable⁵.

Secondly, make sure things are discoverable, simple, and obvious. The example he gave here is that, in some circles, it is “in vogue” to store different environment configuration on different branches in your Git repo. This seems really appealing at first, but it’s actually a huge trap! Not only is it not obvious what branch corresponds to what environment, when it comes to making changes, it’s exceptionally brittle. What if you need to make the same change across all environments? Oh! We’ll just cherry-pick! Except that the commit that you cherry-picked accidentally included some code that isn’t relevant to the new environment, so we have to go revert that. And now it’s not really the same commit anymore, and who knows if it’s going to break when you roll it out. Instead, if you keep things all on your main branch, separated by folders, it’s much easier to understand what code is supposed to be running when and where⁶.

Dan wrote up a blog post version of this talk if you’d like to learn more and don’t want to watch the talk.

10 Things That Super Mario Taught Me About Platform Engineering by Ramiro Berrelleza at Okteto

The last talk I’m going to highlight I’d seen a version of before at the Cloud Native PDX meetup I went to a couple months ago. This talk was not super-technical, but it was a delightful discussion of how to keep our jobs fun. I loved this talk both times I saw it, Ramiro is a really engaging speaker and there’s lots of fun anecdotes and analogies between classic platform games and modern platform teams⁷. I’m just going to spoil his talk here by posting the summary of his “ten things platform teams should do”, but I’d really strongly encourage you to go watch the original!

1. Save them from jumping into cliffs
2. Start easy, then expose the complexity
3. Repeatable patterns let you build a strong foundation
4. Have a clear win after the big boss
5. Ship a sequel so you build a following
6. Let Princess Peach save you from time to time
7. Give your players shortcuts
8. Have your Nintendo Power Hotline available
9. Everyone should have fun
10. If you die, restart!

Introduction to Bootable Containers by Josh Berkus (Red Hat)

OK I know I said “top three” but this one is my honorable mention. Josh’s talk was the last one of the day, and I’m not going to cover all the details, because this post is already too long⁸, but the short version is that bootc is a tool that lets you build bootable virtual machine images from container files⁹. Josh showed how you could use bootc to construct a bootable Kubernetes VM image that you could load onto a bare-metal host, which also handles a bunch of nifty details with rollbacks and failure scenarios. Super-cool tech, would definitely recommend checking it out!

Anyways, that was my summary of my first KCD event—definitely would recommend seeing if there’s one in your area and checking it out¹⁰¹¹! And I’m getting excited to gear up for the full-scale version, i.e., KubeCon in Atlanta this November! I have a bunch of cool stuff lined up in advance of KubeCon that I’m excited to share with you all! But that will have to wait for another post.

As always, thanks for reading.

~drmorr