Something for K8s

Well I got another NAS the other day, so I decided to reconfigure most of my k8s stuff to use that, and hell, why not upgrade it at the same time.

The was about 4 months ago and I just finished last weekend, so here are the steps in case you want to do that in slightly less time.

This won’t be of any use in another three months of course, because everything will change again.

So the things that changed were:

  • as well as moving to a new box, the NAS OS was upgraded, from FreeNAS to TrueNAS,
    moving from FreeBSD to Linux in the process
  • upgraded packer, which also means moving from JSON to HCL
  • upgraded Ubuntu from 16 (!) to 22, which also means moving from preseed config to cloud-init config
  • upgraded MySQL from 5.5 to 8.0, which also means creating native passwords if you want old apps to be able to authenticate
  • upgraded kubernetes from 1.18 to 1.25, which also means updating the API endpoints in the yaml ( and the yaml itself )
  • because I’m publishing this on github, started using vault as a credential store rather than hard-coding things in the yaml / hcl

I realise there’s already an infinite number of these things, but here’s the repository up on github, which has links to one of those boxes-in-boxes architectural diagrams and more READMEs than you can shake a stick at.

It’s called bookcase-ops, as most of this stuff sits in a bookcase just over there *points towards bookcase*.

bookcase-ops
git@github.com:randomnoun/bookcase-ops.git

Tell me more

I guess the lessons learnt are:

packer

  • so you now need to specify all your variables in triplicate
  • the syntax is arbitrarily different
  • it often fails to clean up after failed builds, so if you don’t get it working perfectly the first time, you may have to remove the vmx/vmdks from esx filesystem manually afterwards

ubuntu (in packer)

  • you don’t need a preseed file any more, so chuck that and write a cloud-init file instead
  • don’t let cloud-init partition your drive using lvm, as it’ll just create a 10GB partition by default, requiring manual resizing afterwards
  • the apt package names are different
  • decrease the keystroke interval to save yourself a minute or two whilst it types in the init command
  • start hard-coding your MACs. That may not be necessary either, but I’ve given up faffing about with the network config.

mysql

  • probably want to use native passwords so you don’t need to upgrade everything that connects to mysql

kubernetes

  • try to ignore changes in calico and fabric as much as humanly possible
  • you’ll need to change from freenas-provisioner to democratic-csi
    • which uses UUIDs as PVC identifiers, so make a note of those as things start up so you know where to copy things across
    • because a path like this:
      /mnt/raidvolume/k8s/bnekub02/nfs/volumes/pvc-1839f8c8-b32e-43cb-9346-2ac69079b478
      is apparently more usable than a path like this:
      /mnt/raidvolume/bnekub01-dataset/dev-xwiki/xwiki-pvc

    • also,
      scp -p doesn’t preserve ownership uids and gids, so you probably need to use
      tar --same-owner
  • for your ingress controllers, don’t use this nginx ( which will cause you do go down the MetalLB rabbit hole, which is not worth your time ), use this nginx instead.
  • in your yaml, replace the API endpoints with the new endpoints, some of the networking defaults have changed, and the structure of the nginx backend config has changed
    • e.g. you’ll need to change DNS resolution to ‘Default’, which isn’t the actual default
  • when I was merging the new kube config into my local ~/kube/config file, I had to rename a couple of things due to username collisions.

I guess most people would look at that and say why not use minikube or k3s or something, to which I would say what’s the point of using kubernetes at all if you’re not going to use the unnecessary levels of complicated abstractions it provides.

Add a Comment

Your email address will not be published. Required fields are marked *