Announcing SimKube v1.0!

Ok, folks, this is the one you've all been waiting for¹. I'm finally at a state where I can share some real² results from SimKube. This is going to get split up over at least three posts, because there's a lot of ground to cover, and I want to share as much context and background as I possibly can. Just like my previous series on SimKube, I will update this post with links to the upcoming posts as they're available.

• Part 1 (this post) - Background and Motivation
• Part 2 - Running the Simulations
• Part 3 - Analysis and Following up

I'm pretty excited about this series of posts, to be honest: this is a big milestone for me, and I've been working pretty hard to get here over the last year. I'm also excited because this series marks the release of SimKube v1.0! There's still a lot of work to do on SimKube before it's where I want it to be, but I think SimKube is actually at a point where other people can use it and get useful information out, which is exhilarating³. So with that probably-over-the-top introduction, let's dive in.

Autoscaling Kubernetes #

If you've known me for any length of time, you'll know that I've spent a lot of time in the autoscaling space. When I was at Yelp, I wrote Clusterman⁴ and I think that project was really successful in helping Yelp manage its cluster scaling for a long time. I don't recommend that folks use Clusterman these days, because there are better options out there, but it will always have a soft spot in my heart.

Later on, I became a contributor and reviewer/approver for the Kubernetes Cluster Autoscaler, and even more recently than that, because a contributor to the Karpenter project. So all of that's to say, I've been around this space for a little while (although there are certainly plenty of folks out there who have been doing this longer and better than I have).

In the last few years, there has been a bit of a friendly rivalry that's developed between the two main cluster autoscaling engines, namely the Kubernetes Cluster Autoscaler (KCA) and Karpenter. While these two projects are trying to solve similar (but not exactly the same!) problems, the approaches that they take to solving this problem are very different. KCA is a much older, more mature project, and is well-supported across a host of different ecosystems and cloud providers. Unfortunately for that project, (at least in my opinion), because it was started so early in the Kubernetes project's broader lifecycle, the design makes some assumptions about how users would want to autoscale their clusters that don't hold up in reality⁵. Karpenter, on the other hand, is a much newer autoscaler developed by AWS that strives to use a more modern style of Kubernetes development using controllers. One of Karpenter's goals is to solve some of the problems that users run into with KCA, particularly around scaling speed and node selection. Unfortunately for that project, it's (currently) only available for AWS and Microsoft Azure customers⁶.

There've been a number of prominent KubeCon talks discussing the advantages of Karpenter over KCA, and so, given all of this context and my interest in the autoscaling domain, I thought it would be really interesting to use SimKube to do a head-to-head comparison of the two autoscalers.

Experiment setup #

So how can we use SimKube to compare the Kubernetes Cluster Autoscaler and Karpenter? Well, first we need some input data (or a trace, in the SimKube vernacular) to test on. As I don't currently have access to a large AWS compute cluster running real workloads, I decided to use DeathStarBench to generate some test data. For those who are unfamiliar, DeathStarBench (DSB) is a suite of "production-like" microservice-based applications that are used in a lot of academic literature for testing of distributed systems⁷. It includes a "social network" application, where users can create posts and follow other users in the system, a "media services", where users can write reviews about movies that they've watched, and a "hotel reservations" service, which allows users to book rooms for a fake hotel chain. Each of these applications include helm charts so that they can be installed on Kubernetes, along with a set of scripts that you can use to generate load in different parts of the application.

For the purposes of this experiment, I wasn't too concerned with the specific functionality that the application provided, so I just picked the social network application more-or-less at random. This application has 27 different services to handle things like writing posts, showing the user timeline, etc. In the setup for this experiment, I configured each service to run with CPU request of 1, and a memory request of 1GB^{8 9}: I spun up a single-node Kubernetes cluster on an AWS c6i.8xlarge EC2 instance, which has 32 vCPUs and 64GB of RAM. Then I deployed social network to it, and let it settle into a "steady state" for about 10 minutes, before I started to induce load on the application.

Once the application had been running for a while, I used the scripts provided by DSB to induce load. The first step was to create a social graph; from there, I ran a second script to compose a whole bunch of posts; then I had a whole bunch of users read their home timelines; and then lastly, I had a whole bunch of users read timelines of other users. Here's what it looked like:

The font is a bit small and hard to read on these graphs¹⁰, but what you can hopefully see is that we ran one pod per service (for a total of 27 pods) for the first ~ten minutes, and then we started scaling up. The first bump is when I created the social graph, and the second bump is when I started composing posts. But what happened after that? Why does it stay flat for the rest of the time? Well, here, take a look at this graph:

If you'll recall, I was running this on a single-node Kubernetes cluster with 32 vCPUs, and each pod requests a single vCPU, which means that once 32 pods got scheduled, nothing more could be done. If this had been an actual social network, this would have been a major outage! Fortunately, I do not work at an actual major social network.

First SimKube test: is it a simulator? #

Once I finished the experimental setup, I captured a trace of all the deployments in the system. Before I even got to the autoscaling comparison, there was an important question I needed to answer: is SimKube a simulator? In other words, if I create a simulated environment and replay the trace of the above timeline, the cluster should behave equivalently. I'm happy to say here that the answer seems to be yes! Here are graphs of the pending and running pods for the simulation environment:

To generate these graphs, I replayed the trace I captured 10 times, and I used Prometheus and prom2parquet to save all of the generated metrics from the simulation into parquet files in S3. There are a few quirks in the above data: you can see some odd intermittent jumps and spikes; most of these are, I believe, artifacts of the metrics collection pipeline. You can also see a brief spike of "pending pods" near the beginning of the simulations, as well as a dip in the pending pods at around thirteen minutes in: these differences can be attributed to the fact that my simulation Prometheus was scraping data once per second¹¹, whereas in the original (real) version, I was only scraping once every 30 seconds, so some of these smaller changes are masked in the Grafana charts. Aside from these small differences, at least from a visual inspection, it looks like the simulator is doing exactly the same thing as the "real thing"¹².

"But wait," I hear you say. "Didn't you say you ran this 10 times? I only see one line above, with a couple weird multi-colored bumps in it!" But, actually, you are looking at 10 lines in the above graphs—they're just all right on top of each other! I was actually expecting slightly more variability in the results than actually occurred. I was pleasantly surprised at how close each of the individual simulation runs were to each other.

However, just to demonstrate that these are actually all distinct runs, here is the same set of 10 simulations, showing which pods actually get scheduled (grouped by service). This is where you can start seeing some variability. The pods that get scheduled are essentially random, since it depends on "which pod kube-scheduler sees first", and this is going to be highly susceptible to tiny timing changes. Essentially, if the deployment controller for service A takes a hundredth of a second longer to create a pod than service B in one of the simulations, then the order of A and B's pods in the scheduler queue might get swapped and you'll see different running pods in each simulation¹³.

I know the above plots are tiny, but I'm not interested in the numbers themselves, but in the differences, which are still visible even at a small resolution: for instance, it's quite clear that sim.2 is running a different set of pods about two-thirds of the way through the simulation than some of the other simulations are. For the purposes of this experiment, however, this level of variability is fine. I'm interested in knowing how the autoscalers perform, not exactly which pods get scheduled when. But learning about that will have to wait until the next post!

Wrapping up #

I know this post was just a bunch of setup and I ended you on a cliffhanger. Sorry about that. I hope you still found it interesting! If you would like to take a look at the data I used to write this post, I've made it all publicly available online. This includes the trace that I captured to run these experiments, as well as all the raw Parquet data and the Jupyter notebooks that I did the analysis in. I'd love to hear if you're able to run or replicate these results, and especially if you get different results or have other interesting insights!

I'll be sharing a bunch more about SimKube in a bunch of different venues over the next few months, and there's still a lot of work to be done, so if any of this is interesting to you, I'd love to have your help! Please let me know.

Thanks for reading,

~drmorr

---