Running Docker and Kubernetes on ARM Servers: A Migration Guide

Moving container workloads to ARM servers is mostly straightforward now, which was not true a few years ago. The tooling caught up. What is left is a handful of specific things that will trip you up if nobody warns you about them.

This covers what actually changes when you run Docker and Kubernetes on ARM64, and how to deal with the parts that are genuinely different.

Check your images before anything else

The first question is whether the images you depend on have ARM64 builds. Most popular ones do, published as multi architecture manifests so that docker pull quietly fetches the right variant for the machine you are on.

You can check any image without pulling it:

docker manifest inspect nginx:latest | grep architecture

If you see arm64 in the output, you are fine. Official images for nginx, postgres, redis, mysql, node, python, golang and the rest of the common set all publish ARM64.

Where you are more likely to find gaps is smaller community images, vendor supplied images from commercial software, and anything that has not been updated in a couple of years. Check those specifically rather than assuming.

Building your own images for ARM

If you build images, they are x86 only unless you deliberately made them otherwise. Rebuilding for ARM64 is done with buildx:

docker buildx create --use
docker buildx build --platform linux/amd64,linux/arm64 -t yourorg/yourapp:1.0 --push .

That produces a single tag serving both architectures. Anyone pulling it gets the right one automatically, which means you can migrate gradually rather than switching everything at once.

Two things to know. Building ARM64 images on an x86 machine uses QEMU emulation and is slow, sometimes five to ten times slower than native. If your CI is x86 and your builds start timing out, that is why. Building on an actual ARM machine is dramatically faster, which is one of the quieter arguments for having one in your build fleet.

The other is that base images must be multi arch too. A Dockerfile starting FROM someimage:tag will fail on ARM if that image has no ARM64 variant, no matter how portable your own code is.

The compiled dependency problem

This is where most real friction lives. Interpreted code moves without complaint. Native extensions do not.

In Python, packages like numpy, pandas, psycopg2 and cryptography ship prebuilt wheels. Wheel availability for ARM64 is good now but not universal. Where a wheel is missing, pip falls back to building from source, which needs a compiler and development headers in the image and turns a ten second install into several minutes.

The fix is usually to add build dependencies in a builder stage and copy the result into a slim final image, which is good practice regardless of architecture.

Node modules with native bindings behave the same way. Anything depending on node-gyp will compile rather than download. Go and Rust cross compile cleanly and are rarely a problem.

Java is generally fine. The JVM abstracts the architecture, though any JNI libraries bundled with your application need ARM64 versions.

Kubernetes on ARM

Kubernetes itself runs on ARM64 without special handling. kubeadm, k3s and the managed distributions all support it, and k3s in particular is a common choice on ARM because of its smaller footprint.

The control plane, kubelet, container runtime and CNI plugins all have ARM64 builds. Calico, Flannel and Cilium work.

What needs thought is mixed clusters. If you run ARM and x86 nodes together, and there are good reasons to, you need to make sure pods land on nodes that can run their images. Node affinity handles this:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values: ["arm64"]

Kubernetes sets kubernetes.io/arch automatically, so you do not need to label nodes yourself. Without this, an x86 only image scheduled onto an ARM node produces a crash loop with an exec format error, which is unhelpful until you know what it means.

A mixed cluster is a reasonable migration strategy. Add ARM nodes, pin the workloads you have verified, and move more across as you confirm each one.

Why the core count changes how you size things

An 80 core machine like the Ampere Altra Q80-30 changes the arithmetic of container density. On a 16 core x86 server you are usually constrained by CPU before RAM. On an 80 core box you often hit memory limits first.

This is worth thinking about when setting requests and limits. Requests that made sense on smaller nodes may leave a lot of capacity stranded, because the scheduler packs by request rather than by actual use.

The absence of a shared last level cache helps here too. On x86, one container doing heavy memory work can evict cached data belonging to containers on other cores, and everything slows down together. Ampere cores do not share an L3, so that particular interference does not happen. In practice performance stays more even as density rises.

Pod density per node goes up accordingly, which means fewer nodes, less control plane overhead and a simpler cluster for the same amount of work.

Things that will still bite

Monitoring agents and sidecars are a common oversight. Datadog, New Relic and similar vendors publish ARM64 agents, but if you use something niche, check before migrating rather than after.

Init containers and debug tooling get forgotten because they are not the application. An x86 only busybox or a vendor supplied migration container will fail in exactly the same way as the main image.

Helm charts sometimes pin image tags that only exist for x86, particularly older charts. Read the values file rather than assuming the chart handles it.

Anything requiring CUDA stays on x86 and belongs on a GPU server.

A migration order that works

Start with stateless services that use official base images. Web frontends, API servers, reverse proxies. These almost always move without incident and give you a real result quickly.

Then move workers and queue consumers, which are usually your own code and where you will find any compiled dependency problems in a place that is safe to fail.

Databases last, and only after testing restore procedures on ARM. Postgres and MySQL both run well on ARM64, but a database migration deserves its own plan regardless of architecture.

Keep an x86 node available throughout. Being able to move one workload back while you sort out a dependency is the difference between a calm migration and a stressful one.

Whether it is worth doing

The case is cost per unit of work, not raw speed. Ampere cores are not faster than a high clocked Ryzen core. There are simply a lot more of them, drawing about 3 watts each against 8 to 12 watts for comparable x86 cores, and that shows up in what the machine costs to rent.

If your workload is many small containers, and container workloads usually are, the arithmetic tends to favour ARM. If you run a few large single threaded processes, it does not.

The way to find out is to run one real service on one ARM node for a week and compare against what you pay now. Our Ampere Altra plans start at 233 euro a month with the full 80 cores, which is enough to host a meaningful chunk of a cluster rather than a token test node.