Fixing CreateContainerConfigError in Kubernetes

The first time I hit CreateContainerConfigError I wasted twenty minutes staring at the container image, convinced the registry was flaky. Wrong tree entirely. The image had already pulled. The kubelet had the layers on disk and was ready to start the process. It just couldn't assemble the configuration the container needed, so it refused to hand anything to the runtime.

That distinction is the whole game here. This error happens before the container starts, while the kubelet is stitching together environment variables, mounted files, and command arguments from the objects your pod spec points at. If one of those pointers is wrong, the kubelet stops and parks the pod in CreateContainerConfigError. Nothing runs. There's no crash loop to read, because there was never a running process to crash.

What the status actually means

Kubelet builds a container config object for every container in a pod: env vars, volume mounts, the command and args, resource limits. A lot of that config gets pulled from other Kubernetes objects at start time. An envFrom pulls a whole ConfigMap. A valueFrom.secretKeyRef pulls one key out of a Secret. A volume mounts a ConfigMap as files.

When kubelet resolves those references and one comes back missing, the config build fails and you get this status. Ninety percent of the time it's a ConfigMap or Secret that doesn't exist in the namespace, or a key that isn't in the object you named.

Diagnosis: describe names the culprit

Skip the guessing. kubectl describe tells you exactly what's missing, in plain English, in the events at the bottom.

kubectl describe pod payments-api-7d9f-abcde -n payments

Scroll to Events. You'll see something like:

Warning  Failed  kubelet  Error: configmap "payments-config" not found

or, when the object exists but the key doesn't:

Warning  Failed  kubelet  Error: couldn't find key DB_HOST in ConfigMap payments/payments-config

That second message is gold. It hands you the namespace, the object name, and the exact key. No log spelunking required.

Confirm what's actually there:

kubectl get configmap -n payments
kubectl get configmap payments-config -n payments -o jsonpath='{.data}' | jq
kubectl get secret payments-secrets -n payments -o jsonpath='{.data}' | jq 'keys'

The common causes, and how to fix each

The ConfigMap or Secret doesn't exist

Usually a namespace mismatch or a typo. ConfigMaps and Secrets are namespaced, and a pod can only reference objects in its own namespace. You applied the ConfigMap to default and the deployment lives in payments. Fix it by creating the object in the right place:

kubectl create configmap payments-config \
  --from-literal=DB_HOST=pg.payments.svc \
  -n payments

The referenced key is missing

The object exists, the key doesn't. Here's a broken valueFrom:

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: payments-secrets
        key: DB_URL          # secret actually stores "database-url"

The Secret has database-url, the spec asks for DB_URL, kubelet can't find it, config build fails. Match the key exactly. Casing and hyphens count:

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: payments-secrets
        key: database-url

Same trap with envFrom, which pulls every key at once:

envFrom:
  - configMapRef:
      name: payments-cfg      # object is named "payments-config"

One wrong character and the entire pod won't start. Fix the name to match kubectl get configmap output.

A volume mounts a source that isn't there

The env path gets all the attention, but volumes fail the same way. This mounts a ConfigMap that was never created:

volumes:
  - name: app-config
    configMap:
      name: app-settings      # doesn't exist in this namespace

Kubelet can't build the mount, config fails. Create app-settings, or point at the ConfigMap that does exist.

Optional versus required references

This is the escape hatch nobody reads about. Both configMapKeyRef and secretKeyRef accept optional: true. When it's set, a missing key is silently skipped instead of blocking startup. Useful for config you genuinely might not have in every environment:

env:
  - name: FEATURE_FLAG_ENDPOINT
    valueFrom:
      configMapKeyRef:
        name: feature-flags
        key: endpoint
        optional: true

I'm cautious with this one. Marking a reference optional to make an error go away is how you ship a pod that runs with a blank DATABASE_URL and fails weirdly at runtime instead of loudly at startup. Use optional only when the config is truly optional, not to silence a real misconfiguration.

Image and entrypoint config

Less common, but a bad command or args in the spec can also trip config building, and so can a container that declares no command and inherits no ENTRYPOINT from the image. If your describe output has no ConfigMap or Secret complaint, check whether the container has anything to actually run.

How this differs from CreateContainerError

These two look alike and get conflated constantly. They are not the same failure.

CreateContainerConfigError is a configuration problem. Kubelet couldn't build the config from your references. It never reached the runtime. Fix the ConfigMap, Secret, or volume source.

CreateContainerError is a runtime problem. Config built fine, kubelet handed it to containerd, and the runtime refused to create the container: a bad command path, a name collision, a mount that failed at the OCI layer. The config was valid; the container creation itself broke.

The tell is in the event message. "couldn't find key" or "not found" for a ConfigMap or Secret means Config error, look at your references. Anything about the container runtime, exec paths, or OCI means the plain CreateContainerError, look at the image and command. For the wider map of pod states and where each one bites, our Kubernetes troubleshooting guide lays them out side by side.

The call we'd make

Run kubectl describe pod first, every single time, and read the events before you touch anything. This error is one of the most honest ones Kubernetes gives you. It names the object and the key it couldn't resolve. Diff that against kubectl get configmap and kubectl get secret in the pod's namespace, fix the name, the key, or the namespace, and move on. Reach for optional: true only when the config is genuinely optional. If you're using it to make the red go away, you haven't fixed anything, you've just moved the failure somewhere harder to find.