Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Helm upgrade failed: field is immutable on Service and Job

Some fields cannot be patched at all. Here is which ones, why --force does not help, and the safe recreate procedure for Services, Jobs and StatefulSets.

~5 min readKubernetes
#Helm#Kubernetes#Upgrades#Deployments
A Kubernetes object slab sealed with an amber lock plate while a patch arrow bends away from it

Introduction & Quick Summary

Symptom. A chart upgrade that only changed an image tag fails, and the error names a field rather than a value:

code
Error: UPGRADE FAILED: cannot patch "api" with kind Job: Job.batch "api-migrate" is invalid: spec.template: Invalid value: core.PodTemplateSpec{...}: field is immutable
code
Error: UPGRADE FAILED: failed to replace object: Service "api" is invalid: spec.clusterIP: Invalid value: "": field is immutable

helm list shows the release in failed state, --atomic may already have rolled it back, and helm rollback does not help because the new revision cannot be rendered into a valid object at all.

Cause. Kubernetes treats some fields as immutable after creation: a Service's clusterIP and selector, a Job's spec.template, a StatefulSet's volumeClaimTemplates and serviceName. Helm computes a patch, sends it, and the API server rejects the patch. The chart is not wrong about intent; it is asking for an in-place change the object does not allow.

Fix. Decide which side moves. Either the field is preserved (a Service keeps its allocated cluster IP), or the object is replaced deliberately (a Job gets a new name):

# Service: keep the existing clusterIP by reading it back {{- $svc := lookup "v1" "Service" .Release.Namespace (include "app.fullname" .) }} spec: {{- if and $svc $svc.spec.clusterIP }} clusterIP: {{ $svc.spec.clusterIP }} {{- end }}
# Job: make the name change with the content, so a new object is created metadata: name: {{ include "app.fullname" . }}-migrate-{{ .Values.image.tag | trunc 8 }} annotations: "helm.sh/hook": pre-upgrade "helm.sh/hook-delete-policy": before-hook-creation

The rest of this guide covers which fields reject updates, why --force does not rescue a Job, and the migration procedure for the three objects people hit most.

Core Mechanics, Signatures & Reproducible Setup

Helm's upgrade is a three-way merge: old manifest, new manifest, live object. For most resources that becomes a PATCH, which the API server accepts or rejects field by field. With --force, Helm switches to PUT (a full replace), and a replace still fails on fields the API server considers immutable, so --force changes the error shape rather than removing it.

Reproduce the Service case in a scratch namespace without touching a real chart:

bash
kubectl create namespace immut-demo helm create demo && cd demo helm install demo . -n immut-demo kubectl get svc demo -n immut-demo -o jsonpath='{.spec.clusterIP}{"\n"}' # change the service name template (as a chart refactor would) sed -i 's/^ name: {{ include "demo.fullname" . }}$/ name: {{ include "demo.fullname" . }}-v2/' templates/service.yaml helm upgrade demo . -n immut-demo # Error: UPGRADE FAILED: failed to replace object: Service "demo-v2" is invalid: # spec.clusterIP: Invalid value: "": field is immutable

That last step shows the mechanism precisely: the chart asked for a different object, and Helm tried to replace the old one in place.

The fields that reject updates, and the correct response for each:

# Service: selector and clusterIP are immutable; type is not spec: clusterIP: None # headless is a creation-time decision selector: app.kubernetes.io/name: api # changing this is a new Service
# Job: spec.template is immutable; the whole Job is immutable in practice apiVersion: batch/v1 kind: Job metadata: name: migrate-{{ .Values.image.tag | trunc 8 }} spec: ttlSecondsAfterFinished: 600 template: spec: restartPolicy: Never containers: - name: migrate image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

Key Comparison Table

ObjectImmutable fieldSafe change path
Servicespec.clusterIP, spec.selector, spec.type on some pathsdelete and recreate; or lookup to preserve the IP
Jobspec.template (labels, image, command)new name per revision, or a hook with delete policy
StatefulSetspec.volumeClaimTemplates, spec.serviceName--cascade=orphan, edit, recreate pods
Deploymentspec.selectorrecreate the Deployment, keep the same labels
PVCspec.storageClassName, spec.accessModesnew PVC and a data migration
CronJobspec.schedule is mutable; spec.jobTemplate mostly isrecreate if the API server rejects it
Helm flagWhat it actually doesEffect on immutable fields
(none)three-way PATCHpatch rejected
--forcePUT replace for resources that support itreplace still rejected
--atomicwaits, rolls back on failuremasks the error as a rollback
--cleanup-on-faildeletes newly created resourcesleaves the old object in place
--take-ownershipadopts resources without ownership metadataunrelated

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. Changing a Deployment's selector

The selector is used by the controller to find its pods, so Kubernetes refuses to change it. A chart refactor that renames a label breaks every existing Deployment:

bash
kubectl get deploy demo -n immut-demo -o jsonpath='{.spec.selector.matchLabels}{"\n"}'

The clean sequence keeps the service available and accepts a short period of double-run:

bash
kubectl delete deploy demo -n immut-demo --cascade=orphan helm upgrade demo . -n immut-demo kubectl get rs -n immut-demo # old ReplicaSets become orphans; scale or delete them

--cascade=orphan leaves the pods running, so traffic continues while the new controller adopts new pods. Without it, the delete takes the application down for the length of the rollout.

2. Jobs and the migration that must run once per release

A Job is effectively immutable, and that is a deliberate design: changing the template of a completed Job would mean re-running work under the same name. The two accepted patterns are a new name per revision and a Helm hook:

metadata: name: {{ include "app.fullname" . }}-migrate-{{ .Release.Revision }} annotations: "helm.sh/hook": pre-upgrade "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded

With hook-delete-policy: before-hook-creation, the previous hook Job is removed before the new one is created, so the name can even stay stable. Without the annotation, Helm attempts a patch and the upgrade fails on spec.template.

bash
kubectl get jobs -n immut-demo kubectl logs job/demo-migrate-3 -n immut-demo

3. Preserving an allocated cluster IP

When a Service must keep its IP (a DNS record, a firewall rule, an external-dns entry), the chart should read the live object and echo the value back. lookup returns an empty object on helm template and during a dry run, so guard it:

{{- $existing := lookup "v1" "Service" .Release.Namespace (include "app.fullname" .) }} spec: type: {{ .Values.service.type }} {{- with $existing }} clusterIP: {{ .spec.clusterIP }} {{- end }} ports: - port: {{ .Values.service.port }}

The side effect is that helm template output no longer equals the live manifest. That is acceptable for clusterIP, and it is why teams keep such charts out of GitOps diff checks that expect idempotent output.

4. StatefulSet volume claim templates

volumeClaimTemplates cannot change, because each existing replica already owns a PVC created from the old template. The accepted procedure is to orphan the pods and the PVCs, delete the StatefulSet, and let Helm create it again:

bash
kubectl delete statefulset db -n immut-demo --cascade=orphan helm upgrade db ./db-chart -n immut-demo kubectl get pvc -n immut-demo

The PVCs survive the orphan, so the data does. Pods are recreated one ordinal at a time by the new controller, which is the same rolling behaviour a normal update would have had.

5. When the failure is a rollback, not the patch

With --atomic, the visible error is often a rollback message. The real cause is in the release history:

bash
helm history demo -n immut-demo helm get manifest demo -n immut-demo --revision 2 kubectl describe job -n immut-demo demo-migrate

Read the failed revision's manifest against the live object with kubectl diff -f before retrying. Re-running the same upgrade produces the same rejection and burns the release history.

6. Rendering before you apply

The whole class of errors is cheaper to catch locally: render, then diff against the cluster.

bash
helm template demo . -n immut-demo > /tmp/rendered.yaml kubectl diff -f /tmp/rendered.yaml | head -60

kubectl diff performs a server-side dry run against the live objects, so an immutable-field problem appears before the release exists. In CI, run the same pair on the pull request.

Common Compilation/Runtime Mistakes & Version Gotchas

  • Adding --force as a reflex. It changes the request type; Kubernetes still refuses immutable fields.
  • Re-running the same failing upgrade. Each attempt adds a failed revision, and --atomic hides the reason.
  • Renaming a Service or its selector to "fix" labels. That recreates the object and loses the cluster IP and any static address.
  • Patching a Job's image tag in place. Use a new name or a hook with a delete policy.
  • Editing volumeClaimTemplates and expecting a rolling update. It requires orphaning and recreating.
  • Trusting helm template output for lookup-based charts. It renders without cluster state.
  • Deleting a release to clear a failure. helm uninstall removes the workloads too; use helm rollback or the object-level procedure.
  • Forgetting hook-delete-policy. Hook Jobs accumulate and eventually block the name.

Validate rendered manifests as YAML with the YAML validator, generate a unique suffix for per-revision object names with the UUID generator, and compare the rendered and live manifests with the text diff tool before the next upgrade attempt.

Reference: helm upgrade, Services, and Jobs in the Kubernetes documentation.

Official references

primary documentation

The Kubernetes documentation that defines the behaviour this guide explains.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.