300 alerts for an image that was already there

This morning my phone blew up. Not with one alert, with a flood of them. Alertmanager had grouped them, but the group kept growing. All of them said: KubeJobNotCompleted, namespace pixbin. By the time I had coffee, the count was well past 300.

Nothing about this made sense at first. Pixbin had not been deployed since May(ish) - I was busy with other stuff. The cron jobs that were now failing had run every few minutes for four months without a problem. The only thing that changed was that I pushed a few small commits yesterday and let CI deploy them.

The pods were stuck in ImagePullBackOff. That was strange, because the image was already on the node (I checked). The migration job pulls it first, and the web and worker deployments were happily running it.

The setup

Pixbin runs on a single bare metal NixOS box with Kubernetes on it. The image lives in a private repository on Docker Hub. The Helm chart has a Deployment for the web app, one for the Celery worker, a migration Job, a collectstatic Job, and six CronJobs for cleanup and monitoring tasks. Pull policy is IfNotPresent and every image tag is a git SHA.

The Deployments and the Jobs all had imagePullSecrets. The CronJobs did not. I never noticed, because it never mattered. The migration job runs first on every deploy, pulls the image with credentials, and from then on the image is in the containerd cache. Every other pod on the node, with or without a pull secret, could use it. That worked for a year.

The quick fix

I added imagePullSecrets to all six CronJob templates and deployed. The alerts cleared. Job done, in a sense. But I wanted to know what changed. Adding a secret that “should not be needed” just rubbed me the wrong way. I needed to know what changed that made this suddenly necessary.

My first theory was containerd. The box got a NixOS release upgrade in July and containerd jumped to 2.3. Maybe the new containerd handles cached images from private registries in a different way. This theory was wrong, but it pointed in the right direction.

The error

The kubelet log had the first failure yesterday, exactly at the first cron tick after the deploy finished:

"Failed to pull image" err="failed to pull and unpack image
\"docker.io/denibertovic/pixbin:sha-bb139475\": failed to resolve image:
pull access denied, repository does not exist or may require authorization:
server message: insufficient_scope: authorization failed"

So the kubelet did try to pull. With IfNotPresent and the image sitting in the cache, it should not have. Something made the kubelet decide that the cached image did not count.

The record

On the node, the kubelet root directory had a folder I had never seen before:

/var/lib/kubernetes/image_manager/
  pulled/
  pulling/

The folder was created on July 28. That is the first boot after the NixOS 26.05 upgrade. Inside pulled/ there was one file per image digest, and the ones for pixbin looked like this:

{
  "kind": "ImagePulledRecord",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "lastUpdatedTime": "2026-09-18T13:03:57Z",
  "imageRef": "sha256:4a8578e6...",
  "credentialMapping": {
    "denibertovic/pixbin": {
      "kubernetesSecrets": [
        {
          "uid": "...",
          "namespace": "pixbin",
          "name": "docker-hub",
          "credentialHash": "..."
        }
      ]
    }
  }
}

The kubelet now remembers which credentials pulled which image. That is KEP-2535, Ensure secret pulled images. The feature gate is called KubeletEnsureSecretPulledImages. It was alpha in 1.33, and in Kubernetes 1.35 it went to beta and is on by default.

The rule is simple. If an image was pulled with credentials, a pod that wants to use it must present credentials that match the record. If it cannot, the kubelet does not use the cached image. It pulls again, with whatever the pod has. My cron pods had nothing, so the pull went to Docker Hub anonymously and Docker Hub said no.

I understand now that this closes a real security issue (which I was unknowingly relying on). Before this, any pod on a node could run any private image that some other pod on the same node had pulled. Namespaces did not matter. I’m the only using this “cluster” so it doesn’t really matter but I also manage lost of clusters for work and this could become an issue.

Why it took two months to bite

I did the the upgrade to NixOS 26.05 on July 27. That moved the Kubernetes package from 1.33.3 to 1.36.2. The feature was on from the first boot. So why did the cron jobs keep working until yesterday?

Because of the default policy. The kubelet config option is imagePullCredentialsVerificationPolicy and the default is NeverVerifyPreloadedImages. From the Kubernetes docs:

When the KubeletEnsureSecretPulledImages gets enabled for the first time, either by a kubelet upgrade or by explicitly enabling the feature, if a kubelet is able to access any images at that time, these will all be considered pre-pulled. This happens because in this case the kubelet has no records about the images being pulled.

The image from the May deploy was pulled by the old kubelet. The new kubelet had no record for it. No record means “preloaded”, and preloaded images are never verified. So the cron jobs ran that image for two months just fine.

Then I deployed. The migration job pulled the new image with the docker-hub secret. The new kubelet wrote a record. Two hours later the first cron job asked for the same image with no credentials, and the kubelet said: not yours.

Containerd 2.3 had nothing to do with it. The pull was a kubelet decision.

Lessons

Every pod that uses a private image needs its own pull secret. Deployments, Jobs, CronJobs, and the one-off debug pods I start with kubectl run. I used to think of the pull secret as a per-node thing. It is a per-pod thing now. This is the correct approach and I’m happy it’s turned on by default.

The upgrade did not break anything on the day it happened. It armed a trap that the next deploy triggered. My instinct was to look at what I changed in the deploy. I should have also asked what changed on the platform since the deploy before that one. Two months of “it works” told me nothing, because nothing had exercised the new code path.

Read the kubelet log, not only the pod events. The pod event said ImagePullBackOff. The kubelet log said the same, but with a timestamp that lined up exactly with the first cron tick after the deploy. That timing was the clue.

Set failedJobsHistoryLimit on CronJobs. I also set successfulJobsHistoryLimit: 0 and failedJobsHistoryLimit: 1 in the same commit. A cron job that fires every few minutes buried me in failed pods and alerts. 300 alerts for one root cause is not useful information. It also caused a second alert: KubeletTooManyPods - new pods could not be scheduled anymore. I will consider increasing the max pod limit on the kubelet since there’s plenty of room - it’s a beefy node.

I need to check my other charts. I have the same chart layout in a few other projects, and all of them run on the same box. Two of them were last deployed before the upgrade, so they may fall for this trap. They are next.

I need to remember this when I rotate the registry credentials. The record stores a hash of the credentials. New credentials do not match old records, so the first pod after a rotation triggers a real pull. That is fine as long as every pod has the new secret. If I forget one, I get another early morning.