Analyzing SimKube 1.0: How well does it work?

Ok, this post is the last part of my 3-part series on SimKube 1.0. Over the last couple of weeks, we walked through some background about SimKube, the Kubernetes Cluster Autoscaler (KCA) and Karpenter, and then performed an in-depth analysis of the two autoscalers based on simulation data. Today, we're going to take a look at SimKube itself: its performance, some lessons that I learned from this set of experiments, and next steps in the Kubernetes simulation world. If you want to go back and read the previous posts from this series first (recommended), you can find them here:

• Part 1 - Background and Motivation
• Part 2 - Running the Simulations
• Part 3 (this post) - Analysis and Following up

As before, I'll note that all of the raw data for these experiments is publicly available for download here if you're interested in trying it out on your own!

Who watches the Simulator? #

One of the selling points of SimKube is that you can (supposedly) run simulations of multi-thousand-node Kubernetes clusters on your laptop. But, all the experiments I ran last week were run on an AWS c6i.8xlarge instance with 32 vCPUs and 64GB of RAM! I dunno about you, but my laptop doesn't have 32 vCPUs and 64GB of RAM, so what gives?

As I hinted at briefly last week, it all comes down to metrics. SimKube itself actually ran fine, even with my largest simulations. I'm not going to include the graphs here because they're not that interesting, but both the simulation controller and the simulation driver pods used a tiny fraction of the host's available CPU and about 20MB of memory. The KWOK controller¹ similarly uses around half a core and 60-100MB of RAM; we only had about a maximum of 100 nodes in the Karpenter experiment, and more like 1000 nodes in the CA experiment². If we assume that this resource consumption scales linearly³ then you're looking at maybe half a gig of memory for a 5000-node cluster, which is the largest size supported by "official" Kubernetes (again I don't think these graphs are particularly interesting, you can see them in the Jupyter notebooks for the experiments)

Just for funsies, we can also take a look at the resource utilization for the Kubernetes control plane (apiserver, controller-manager, and scheduler). We're not taxing Kubernetes itself particularly hard with this experiment, but this can give you an idea of the resource utilization by the k8s control plane for a middling-size cluster:

These graphs are from the set of KCA experiments, which had a larger number of nodes, and thus are likely to put higher load on the control plane. You can see that the apiserver is the most heavily loaded, using 3-4 cores during periods of intense scaling activity, and around 2GB of RAM⁴. The above three graphs probably look pretty similar to stuff you have internally on Grafana or SignalFX or whatever monitoring tool you use⁵, so maybe not that interesting⁶, except to confirm that the Kubernetes control plane performs somewhat similarly in a simulated environment as a "real" environment⁷. The graphs for the Karpenter simulations look similar, but slightly lower resource utilization.

Ok, so SimKube itself doesn't seem like it's using that many resources, what's going on that requires a giant EC2 inst—

Oh. Oh my. What is going on here???

These graphs are showing the resource utilization for our metrics pipeline. If you recall, I'm using the standard⁸ kube-prometheus stack to collect metrics, and I'm using prom2parquet⁹ to export those metrics to an efficient¹⁰ storage format on S3. Also if you recall, I made the somewhat dubious decision to scrape metrics every second. So, what you can see here is that my Prometheus pod is using a solid 2-3 cores and a gradually-increasing-but-lets-call-it-5GB of RAM over continuously over the course of the simulation, and prom2parquet is using more like 6-10 cores and 30-ish GB of RAM over the course of the simulation! And now you see why I ran this on a fairly hefty EC2 instance instead of on my laptop.

Wait but how did you even analyze this data if there's so much of it? #

Ok, let's get one thing out of the way first: this is not Big Data™️. A few hundred gigs of data stored in an S3 bucket isn't even close to Big Data™️. But look, it's still more than I can fit in my laptop's memory, and I'd like to get some useful results out of this thing, so, like, how did I make all these pretty graphs?

Early on in the process, when I was just dealing with a couple gigs of data (i.e., for the smaller experiments), I was downloading the Parquet files locally, caching them for easy re-access later, and then using Pandas to munge the data¹¹. (I'm also using Bokeh to produce the graphs, which is the least-worst Python plotting utility out there). Pandas is…fine. It's got a weird programming model, it uses a bunch of non-standard conventions and you have to learn what all the types mean, and I feel like I forget how it works every few months and have to go relearn it again, but, you know, whatever. It's fine.

Anyways, I packaged all of my analysis tools up into a wrapper suite called DataKube and used that to do all my processing, and it worked pretty well until I tried to process the several hundred gee-bees from my larger experiments and promptly crashed my laptop.

After banging my head into a wall for a couple days, I took a step back to analyze my options. The nice thing about using Parquet as my storage format is that there are a bunch of data analytics platforms and tools out there that can work on the data. As far as I can tell¹², there are roughly four options here:

1. You can spin up a really beefy EC2 instance to do your data analysis, if you feel like giving AWS even more money than you already are.
2. You can use AWS Athena to run queries against your Parquet files that you've nicely stored in S3, if you feel like giving AWS even more money than you already are.
3. You can pay some third-party vendor to analyze your Parquet files for you¹³.
4. Or, you can use DuckDB, which is free and open-source.

After carefully considering my options and my bank account, I elected for option 4¹⁴.

So then I took two weeks to completely rewrite DataKube using SQL and DuckDB, I was finally able to do the analysis on these bigger simulations. I was really pleasantly surprised to discover that DuckDB is fast. Like, really fast. Even on the small simulation runs I started out with, Pandas took 30-40 seconds to crunch all the data into a nice digestible format; DuckDB did it basically instantaneously. And for the large simulation runs, Pandas couldn't even finish crunching the data, but DuckDB does it in about 10 minutes¹⁵. So yea—I can definitely say that, if you're trying to do this sort of analysis on your own, DuckDB is the way to go¹⁶.

Putting a bow on it #

So there you have it, folks: a complete series on using SimKube to solve some "real" problems! I hope you enjoyed this series of posts, and I would love to hear from you if you have used SimKube or you want to use SimKube to help understand your own infrastructure better!

These last three blog posts required a significantly higher level of effort than the usual schlock that I throw up here and call writing, so I'm honestly probably going to take a break here for a few weeks. I definitely have a lot more things to say, and some articles will be coming out in some other venues soon, so don't go anywhere! Thanks to all of my subscribers for reading and supporting the work that I'm doing at Applied Computing—it's definitely appreciated.

Till next time,

~drmorr

---