Helm upgrade failed: field is immutable on Service and Job
Some fields cannot be patched at all. Here is which ones, why --force does not help, and the safe recreate procedure for Services, Jobs and StatefulSets.

Introduction & Quick Summary
Symptom. A chart upgrade that only changed an image tag fails, and the error names a field rather than a value:
Error: UPGRADE FAILED: cannot patch "api" with kind Job: Job.batch "api-migrate" is invalid:
spec.template: Invalid value: core.PodTemplateSpec{...}: field is immutableError: UPGRADE FAILED: failed to replace object: Service "api" is invalid:
spec.clusterIP: Invalid value: "": field is immutablehelm list shows the release in failed state, --atomic may already have rolled it back, and helm rollback does not help because the new revision cannot be rendered into a valid object at all.
Cause. Kubernetes treats some fields as immutable after creation: a Service's clusterIP and selector, a Job's spec.template, a StatefulSet's volumeClaimTemplates and serviceName. Helm computes a patch, sends it, and the API server rejects the patch. The chart is not wrong about intent; it is asking for an in-place change the object does not allow.
Fix. Decide which side moves. Either the field is preserved (a Service keeps its allocated cluster IP), or the object is replaced deliberately (a Job gets a new name):
# Service: keep the existing clusterIP by reading it back
{{- $svc := lookup "v1" "Service" .Release.Namespace (include "app.fullname" .) }}
spec:
{{- if and $svc $svc.spec.clusterIP }}
clusterIP: {{ $svc.spec.clusterIP }}
{{- end }}# Job: make the name change with the content, so a new object is created
metadata:
name: {{ include "app.fullname" . }}-migrate-{{ .Values.image.tag | trunc 8 }}
annotations:
"helm.sh/hook": pre-upgrade
"helm.sh/hook-delete-policy": before-hook-creationThe rest of this guide covers which fields reject updates, why --force does not rescue a Job, and the migration procedure for the three objects people hit most.
Core Mechanics, Signatures & Reproducible Setup
Helm's upgrade is a three-way merge: old manifest, new manifest, live object. For most resources that becomes a PATCH, which the API server accepts or rejects field by field. With --force, Helm switches to PUT (a full replace), and a replace still fails on fields the API server considers immutable, so --force changes the error shape rather than removing it.
Reproduce the Service case in a scratch namespace without touching a real chart:
kubectl create namespace immut-demo
helm create demo && cd demo
helm install demo . -n immut-demo
kubectl get svc demo -n immut-demo -o jsonpath='{.spec.clusterIP}{"\n"}'
# change the service name template (as a chart refactor would)
sed -i 's/^ name: {{ include "demo.fullname" . }}$/ name: {{ include "demo.fullname" . }}-v2/' templates/service.yaml
helm upgrade demo . -n immut-demo
# Error: UPGRADE FAILED: failed to replace object: Service "demo-v2" is invalid:
# spec.clusterIP: Invalid value: "": field is immutableThat last step shows the mechanism precisely: the chart asked for a different object, and Helm tried to replace the old one in place.
The fields that reject updates, and the correct response for each:
# Service: selector and clusterIP are immutable; type is not
spec:
clusterIP: None # headless is a creation-time decision
selector:
app.kubernetes.io/name: api # changing this is a new Service# Job: spec.template is immutable; the whole Job is immutable in practice
apiVersion: batch/v1
kind: Job
metadata:
name: migrate-{{ .Values.image.tag | trunc 8 }}
spec:
ttlSecondsAfterFinished: 600
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"Key Comparison Table
| Object | Immutable field | Safe change path |
|---|---|---|
| Service | spec.clusterIP, spec.selector, spec.type on some paths | delete and recreate; or lookup to preserve the IP |
| Job | spec.template (labels, image, command) | new name per revision, or a hook with delete policy |
| StatefulSet | spec.volumeClaimTemplates, spec.serviceName | --cascade=orphan, edit, recreate pods |
| Deployment | spec.selector | recreate the Deployment, keep the same labels |
| PVC | spec.storageClassName, spec.accessModes | new PVC and a data migration |
| CronJob | spec.schedule is mutable; spec.jobTemplate mostly is | recreate if the API server rejects it |
| Helm flag | What it actually does | Effect on immutable fields |
|---|---|---|
| (none) | three-way PATCH | patch rejected |
--force | PUT replace for resources that support it | replace still rejected |
--atomic | waits, rolls back on failure | masks the error as a rollback |
--cleanup-on-fail | deletes newly created resources | leaves the old object in place |
--take-ownership | adopts resources without ownership metadata | unrelated |
Practical Edge Cases, Error Fixes & Deep Dive Code Examples
1. Changing a Deployment's selector
The selector is used by the controller to find its pods, so Kubernetes refuses to change it. A chart refactor that renames a label breaks every existing Deployment:
kubectl get deploy demo -n immut-demo -o jsonpath='{.spec.selector.matchLabels}{"\n"}'The clean sequence keeps the service available and accepts a short period of double-run:
kubectl delete deploy demo -n immut-demo --cascade=orphan
helm upgrade demo . -n immut-demo
kubectl get rs -n immut-demo # old ReplicaSets become orphans; scale or delete them--cascade=orphan leaves the pods running, so traffic continues while the new controller adopts new pods. Without it, the delete takes the application down for the length of the rollout.
2. Jobs and the migration that must run once per release
A Job is effectively immutable, and that is a deliberate design: changing the template of a completed Job would mean re-running work under the same name. The two accepted patterns are a new name per revision and a Helm hook:
metadata:
name: {{ include "app.fullname" . }}-migrate-{{ .Release.Revision }}
annotations:
"helm.sh/hook": pre-upgrade
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeededWith hook-delete-policy: before-hook-creation, the previous hook Job is removed before the new one is created, so the name can even stay stable. Without the annotation, Helm attempts a patch and the upgrade fails on spec.template.
kubectl get jobs -n immut-demo
kubectl logs job/demo-migrate-3 -n immut-demo3. Preserving an allocated cluster IP
When a Service must keep its IP (a DNS record, a firewall rule, an external-dns entry), the chart should read the live object and echo the value back. lookup returns an empty object on helm template and during a dry run, so guard it:
{{- $existing := lookup "v1" "Service" .Release.Namespace (include "app.fullname" .) }}
spec:
type: {{ .Values.service.type }}
{{- with $existing }}
clusterIP: {{ .spec.clusterIP }}
{{- end }}
ports:
- port: {{ .Values.service.port }}The side effect is that helm template output no longer equals the live manifest. That is acceptable for clusterIP, and it is why teams keep such charts out of GitOps diff checks that expect idempotent output.
4. StatefulSet volume claim templates
volumeClaimTemplates cannot change, because each existing replica already owns a PVC created from the old template. The accepted procedure is to orphan the pods and the PVCs, delete the StatefulSet, and let Helm create it again:
kubectl delete statefulset db -n immut-demo --cascade=orphan
helm upgrade db ./db-chart -n immut-demo
kubectl get pvc -n immut-demoThe PVCs survive the orphan, so the data does. Pods are recreated one ordinal at a time by the new controller, which is the same rolling behaviour a normal update would have had.
5. When the failure is a rollback, not the patch
With --atomic, the visible error is often a rollback message. The real cause is in the release history:
helm history demo -n immut-demo
helm get manifest demo -n immut-demo --revision 2
kubectl describe job -n immut-demo demo-migrateRead the failed revision's manifest against the live object with kubectl diff -f before retrying. Re-running the same upgrade produces the same rejection and burns the release history.
6. Rendering before you apply
The whole class of errors is cheaper to catch locally: render, then diff against the cluster.
helm template demo . -n immut-demo > /tmp/rendered.yaml
kubectl diff -f /tmp/rendered.yaml | head -60kubectl diff performs a server-side dry run against the live objects, so an immutable-field problem appears before the release exists. In CI, run the same pair on the pull request.
Common Compilation/Runtime Mistakes & Version Gotchas
- Adding
--forceas a reflex. It changes the request type; Kubernetes still refuses immutable fields. - Re-running the same failing upgrade. Each attempt adds a failed revision, and
--atomichides the reason. - Renaming a Service or its selector to "fix" labels. That recreates the object and loses the cluster IP and any static address.
- Patching a Job's image tag in place. Use a new name or a hook with a delete policy.
- Editing
volumeClaimTemplatesand expecting a rolling update. It requires orphaning and recreating. - Trusting
helm templateoutput forlookup-based charts. It renders without cluster state. - Deleting a release to clear a failure.
helm uninstallremoves the workloads too; usehelm rollbackor the object-level procedure. - Forgetting
hook-delete-policy. Hook Jobs accumulate and eventually block the name.
Validate rendered manifests as YAML with the YAML validator, generate a unique suffix for per-revision object names with the UUID generator, and compare the rendered and live manifests with the text diff tool before the next upgrade attempt.
Reference: helm upgrade, Services, and Jobs in the Kubernetes documentation.
Official references
primary documentationThe Kubernetes documentation that defines the behaviour this guide explains.
Try matching browser tools
All toolsSmall utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Kubernetes guides
All Kubernetes guidesKubernetes 1.35: kubelet will not start on a cgroup v1 node
A 1.35 kubelet refuses a cgroup v1 host, and the removed pod-infra flag stops custom images. Check the kernel command line before the node pool.
Helm: another operation (install/upgrade/rollback) is in progress
Helm 3 stores the lock inside the release secret. Read the pending status, roll back to the last good revision, or clear the record deliberately.
Kubernetes Multi-Attach error: the volume is still attached elsewhere
One claim, two nodes. Find the current holder, clear a stale attachment after a node failure, and give the volume a single writer.
Kubernetes: Pod stuck in Terminating (finalizers and grace periods)
A pod disappears only when finalizers clear and containers stop. Read deletionTimestamp, find the blocker, and force-delete only a dead node.
More in Kubernetes
All Kubernetes guidesThe guides filed next to this one in Kubernetes, in library order.