It's Memorial Day here in the USA My family celebrated it together yesterday so my plan for the day was to "lift and shift" my personal Kubernetes cluster from one cluster to another MOSTLY because I wanted to upgrade FluxCD.

I was under the incorrect assumption that doing an in-place upgrade of Flux was difficult, if not impossible, without some severe disruptions to the services running on that cluster.

I was mostly thinking it would be hard to upgrade the various flux manifests from one CRD version to the next and especially hard to upgrade around the ingress-nginx I have installed WITHOUT it dropping the load balancer and forcing me to repoint all of my DNS.

Considering I was prepared to rebuild all of my services from scratch in the new cluster, I realized this was a perfect time to attempt an in-place upgrade to see what sort of problems do arise. Worst case I wasted a few minutes and proceeded with my rebuild.

Luckily, I was wrong, and it took me all of 3 minutes!

Upgrading Flux

I previously searched around a couple of times trying to find any documentation or blog posts on how to upgrade Flux. I mostly found older posts related to upgrading various v1 situations and this was v2. In a last ditch effort, I tried a couple of different searches today and finally ran across this Github Discussion.

If you installed it using flux bootstrap, upgrading should be as easy as running that again with the right repository flags...

Wait what? I can just re-run the flux bootstrap command with a newer version of the cli tool and it just works?!?!?!

I'm pleased to say it does!

All I had to do was re-run the exact same flux bootstrap ... command I had used previously which was:

$ flux bootstrap github \
  --components-extra=image-reflector-controller,image-automation-controller \
  --owner=frankwiles \
  --repository=personal-ops \
  --branch=main \
  --path=cluster/ \
  --read-write-key \
  --personal

After it spit out some progress output of what it was doing all of the containers kicked over and boom, all done.

Before Flux v0.41.2

$ flux version
flux: v0.41.2
helm-controller: v0.31.2
image-automation-controller: v0.31.0
image-reflector-controller: v0.26.1
kustomize-controller: v0.35.1
notification-controller: v0.33.0
source-controller: v0.36.1

v0.41.2 was the last "v2"-ish release prior to them starting the 2.0 release candidates. So while conceptually "v2", I was actually running a really odd version to attempt this with.

After Flux Upgrade

$ flux version
flux: v2.3.0
distribution: flux-v2.3.0
helm-controller: v1.0.1
image-automation-controller: v0.38.0
image-reflector-controller: v0.32.0
kustomize-controller: v1.3.0
notification-controller: v1.3.0
source-controller: v1.3.0

Quite the shift of version numbers!

Why use FluxCD?

I mentioned how easy this went on social media and that I was going to use the time saved to write up this blog post. I got a few questions about why we use Flux vs ArgoCD or why we use these tools at all.

When we first started using Kubernetes with our clients, it was common to simply give your CI systems admin level k8s credentials and have them run kubectl apply or helm upgrade commands to initiate deploys. This never set well with us as the security of most CI systems is usually an after thought. In most cases we deployed our own private CI runners INSIDE our k8s clusters to avoid the bulk of this attack vector, but it was a pain.

We then found a long abandoned project, that I forget the name of, but was very similar to Flux. When it was obvious that tool wasn't going to continue we made the switch.

What tools like Flux and Argo do for you is they provide true gitops yaml manifests that you change to affect changes in your Kubernetes clusters.

They run services inside your cluster and watch a specific git repository looking for changes to these manifests. Let's say we have a repo named gitops and a cluster named bob, the flow is essentially:

• Make a change to a YAML manifest in the gitops repo, say changing a Docker image from v9.0.1 to v9.0.2. Doesn't really matter what the change is, just that there is some change you want to have happen in your cluster.
• The repo is configured to fire a webhook that alerts the CD system (Flux or Argo) that there is a change. This may run inside of bob or in the case of ArgoCD can actually be a centralized Argo service that powers several clusters.
• The CD containers perform a git pull and reconciles the change.
• Assuming there aren't any errors, like a YAML syntax error or something, it applies the changes inside of your bob cluster.

Benefits of CD and gitops in a Kubernetes landscape are:

• No cluster credentials need to exist in your CI systems. All communication of the intent of your change happens via a git repo. While an attacker could still wreck havoc on your cluster by removing services or adjusting security configurations they would not have direct access to the cluster.
• Developers can also make changes directly without the need for k8s access or knowledge of kubectl and helm.
• Changes can be vetted using Pull Requests.
• Changes are tracked in the git commit history for posterity.

Lessons Learned

You should always do one more search before embarking on a long task that seems like it should be easier than it first appears. You might get lucky like I did!

Also always write down the exact flux bootstrap ... command you used when bringing up a new cluster. Personally, I like to document this in the README.md at the root of the gitops repository I'm using for that cluster.

Hopefully this helps other people confused by how to easily upgrade FluxCD. If your company struggles with Kubernetes or gitops you might reach out as we help many of our clients with Kubernetes infrastructure and operations needs including being your entire ops staff if that makes sense.