Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

kubectl apply: metadata.annotations: Too long: must have at most 262144 bytes

Client-side apply copies the manifest into an annotation. Use server-side apply, remove the legacy annotation, and resolve field conflicts once.

~5 min readKubernetes
#Kubernetes#kubectl#CRD#DevOps
A Kubernetes manifest crate with an oversized annotation strip, cyan server side apply gate and amber size dial

Introduction & Quick Summary

text
The CustomResourceDefinition "widgets.example.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

The API server accepts at most 256 KiB of annotations per object, and kubectl apply adds one annotation that regularly reaches that size: kubectl.kubernetes.io/last-applied-configuration, a full copy of the manifest you applied. Large CRDs, big ConfigMaps, and any generated manifest copied from helm template output blow through the limit.

Four causes and their fixes:

  • Client-side apply stores the whole manifest. Switch to --server-side, which tracks fields without an annotation.
  • The annotation is already there and every later apply grows it. Remove it once, then apply server-side.
  • The manifest is large by nature. A CRD with a huge openAPIV3Schema or a ConfigMap of configuration data will exceed 256 KiB in that annotation regardless of formatting.
  • kubectl replace is not the same fix. It works on objects small enough for etcd, but it drops the annotation only if the object is recreated or the annotation is deleted first.
SymptomObjectCauseFix
metadata.annotations: Too long on a CRDCRDSchema copy in last-applied-configurationkubectl apply --server-side
Same error on a ConfigMapConfigMapLarge data block--server-side, or create configmap --from-file
Error appears only after the second applyAnyAnnotation from the first applyDelete the annotation, then server-side apply
Error after a Helm-generated applyAnyhelm template | kubectl applyUse Helm, or server-side apply

Core Mechanics, Signatures & Reproducible Setup

Client-side apply computes a three-way merge in kubectl: the last applied manifest, the live object, and the new manifest. To make that possible it writes the applied manifest into the kubectl.kubernetes.io/last-applied-configuration annotation. Annotations live in object metadata and are stored in etcd with the object, and the API server enforces a 256 KiB ceiling on the total annotation size.

Server-side apply (SSA) moves the merge to the API server. Field ownership is tracked in managedFields, which is structured metadata rather than a copy of the document, so the size limit stops being a problem for ordinary manifests.

A reproduction needs nothing more than a CRD with a large schema:

bash
# a CRD whose schema is big enough to matter python3 - <<'PY' import json props = {f"field{i}": {"type": "string", "description": "x" * 400} for i in range(600)} crd = { "apiVersion": "apiextensions.k8s.io/v1", "kind": "CustomResourceDefinition", "metadata": {"name": "widgets.example.com"}, "spec": { "group": "example.com", "scope": "Namespaced", "names": {"plural": "widgets", "singular": "widget", "kind": "Widget"}, "versions": [{ "name": "v1", "served": True, "storage": True, "schema": {"openAPIV3Schema": {"type": "object", "properties": props}} }] } } open("crd.json", "w").write(json.dumps(crd)) PY kubectl apply -f crd.json
text
The CustomResourceDefinition "widgets.example.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
bash
kubectl apply --server-side -f crd.json # customresourcedefinition.apiextensions.k8s.io/widgets.example.com serverside-applied

Key Comparison Table

The apply mode decides whether the manifest is copied into the object at all.

ModeWhere the merge happensStores last appliedAnnotation limit relevantField conflicts
kubectl applyClientYes, full manifestYesLast writer wins
kubectl apply --server-sideAPI serverNoNoReported, --force-conflicts to take over
kubectl replaceClient, whole objectNoNoObject replaced wholesale
kubectl create --save-configClientYesYesCreate only
Helm 3Client, release storageNo, stored in a SecretNoHelm owns fields

Removing the existing annotation is sometimes enough for one more apply, but the annotation returns unless the apply mode changes.

ActionRemoves annotationFuture applies stay smallRisk
kubectl annotate ... last-applied-configuration-YesNoNext apply recreates it
kubectl apply --server-sideYesYesField conflicts need review
kubectl replace -fYes, if absent from the fileYes, until an apply runsReplaces the whole object
kubectl delete then createYesYes, until an apply runsDowntime and finalizers

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

Adopt objects into server-side apply

SSA refuses to take ownership of fields another manager set, and the error names the conflicting manager and field. That is expected during migration, not a failure.

bash
kubectl apply --server-side -f crd.json # Apply failed with 1 conflict: conflict with "kubectl-client-side-apply": .spec.versions
bash
# review, then take ownership deliberately kubectl apply --server-side --force-conflicts -f crd.json kubectl get crd widgets.example.com -o jsonpath='{.metadata.managedFields[*].manager}{"\n"}'

--force-conflicts is a one-time migration step. Leaving it in CI hides real ownership mistakes, because two automation systems will keep overwriting each other silently.

Strip the legacy annotation first

If the object is already too large to be updated by the old annotation, delete the annotation with a JSON patch rather than another apply.

bash
kubectl annotate crd widgets.example.com kubectl.kubernetes.io/last-applied-configuration-
bash
# equivalent, and safer for scripted runs: patch with a null value kubectl patch crd widgets.example.com --type=json \ -p='[{"op":"remove","path":"/metadata/annotations/kubectl.kubernetes.io~1last-applied-configuration"}]'

The ~1 escape is the JSON Pointer form of / inside a key, and a patch that targets the wrong key changes nothing.

ConfigMaps and large data blocks

The same limit applies to any object, and a ConfigMap built from many files reaches it easily. kubectl create configmap --from-file writes data without the annotation, which makes it a practical workaround in scripts.

bash
kubectl create configmap app-config --from-file=./config --dry-run=client -o yaml > cm.yaml kubectl apply --server-side -f cm.yaml
bash
# confirm the annotation is the reason, not object size kubectl get cm app-config -o json | jq '.metadata.annotations | map_values(length)'

A ConfigMap near 1 MiB also risks the etcd request limit, which is a different error (etcdserver: request is too large) and needs the data split across objects or moved to a Secret or a volume from a file.

CRDs with very large schemas

A CRD schema of a few hundred kilobytes is legitimate, but it makes the client-side annotation path unusable. Server-side apply handles it, and kubectl replace is an alternative for one-off installs.

bash
kubectl replace -f crd.json kubectl get crd widgets.example.com -o jsonpath='{.metadata.annotations}{"\n"}'
bash
# check how much room the schema takes before it fails wc -c crd.yaml kubectl explain widget.spec --api-version=example.com/v1 | head -20

If the CRD must stay compatible with tooling that only does client-side apply, split the schema into $ref-free but smaller definitions, or accept that installation requires server-side apply.

Helm output applied with kubectl

helm install stores release data in a Secret and does not hit this limit, but helm template | kubectl apply -f - is client-side apply and does.

bash
# keep Helm as the manager helm upgrade --install app ./chart # or render once and apply server-side helm template app ./chart | kubectl apply --server-side -f -

Mixing managers on the same object produces field conflicts later, so pick one per object and keep it.

Managed fields growth in old clusters

SSA records every manager in metadata.managedFields, which grows with each controller that touches the object. On clusters older than 1.22 the field was not pruned the way it is now, and a long-lived object can accumulate entries.

bash
kubectl get deploy app -o json | jq '.metadata.managedFields | length' kubectl apply --server-side --field-manager=platform-cli --force-conflicts -f deploy.yaml

A stable --field-manager per automation system keeps the list short and makes conflicts traceable.

Common Compilation/Runtime Mistakes & Version Gotchas

Reapplying the same manifest after the error. The annotation already exists and the new apply rewrites it, so the failure repeats at the same size. Delete the annotation or change the mode.

Using --validate=false. It skips schema validation, not the annotation limit, so the error remains and the object is accepted in an invalid state if it fits.

Blaming YAML formatting. Comments and indentation are stripped by the parser, so reformatting saves nothing; the copied manifest is serialized JSON in the annotation.

Assuming kubectl create avoids the problem. create --save-config adds the same annotation. Plain create without --save-config does not, but it cannot update an existing object.

Forgetting that CRDs are cluster-scoped. kubectl annotate without -n targets the cluster scope, which is correct for a CRD and wrong for a namespaced object.

Version notes. Server-side apply became GA in Kubernetes 1.22 and --server-side has been available since 1.16. managedFields is beta by default from 1.18 and on by default in current releases. The 262144-byte limit has been enforced for many releases, and the message names the object kind, so a CRD and a ConfigMap error have the same cause.

Order of operations: try kubectl apply --server-side first, check conflicts with --dry-run=server, take ownership with --force-conflicts once. When a manifest is too large to read by hand, the YAML formatter makes the structure reviewable, and the text diff shows exactly what a generated manifest added between two versions.

Official references: Annotations and Server-side apply.

Official references

primary documentation

The Kubernetes documentation that defines the behaviour this guide explains.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.