Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes Jobs and CronJobs: Completions, Parallelism, Backoff, and Missed Schedules

completions counts successes, parallelism counts concurrency, and backoffLimit counts retries. Plus podFailurePolicy, restartPolicy differences, and the CronJob timezone trap.

~5 min readKubernetes
#Kubernetes#Jobs#CronJob#Batch
3D illustration of parallel batch task cubes reaching completion checkmarks under a schedule clock

Quick Summary

A Job runs pods until a required number of them exit successfully; a CronJob creates Jobs on a schedule. The counters are explicit: completions is how many successful pod exits finish the Job, parallelism is how many run at once, and backoffLimit is how many failures are tolerated before the Job is marked Failed.

restartPolicy decides who retries. With Never, a failed pod stays behind and the Job controller creates a replacement, so pod count grows and logs of the failed attempt survive. With OnFailure, the kubelet restarts the container inside the same pod, so backoffLimit counts container restarts and the pod list stays small. Neither policy retries after a successful exit; that is the contract that makes a Job terminable at all.

A CronJob adds scheduling on top: a five-field cron expression, a timezone, concurrencyPolicy for overlapping runs, startingDeadlineSeconds for late starts, and history limits that control how many finished Jobs linger. The failure that bites most teams is not cron syntax but a missed schedule evaluated against the controller's last-seen time, and jobs that overlap because a previous run had not finished.

Basic Syntax & Core Signatures

yaml
apiVersion: batch/v1 kind: Job metadata: name: reindex-20261115 spec: completions: 4 # successful pods required (default 1) parallelism: 2 # pods running simultaneously (default 1) backoffLimit: 3 # retries before Failed (default 6) activeDeadlineSeconds: 1800 # wall clock cap; exceeded => Failed ttlSecondsAfterFinished: 3600 # delete the Job an hour after it finishes completionMode: Indexed # Indexed | NonIndexed podFailurePolicy: # optional: fail fast on specific exits rules: - action: Fail onExitCodes: operator: In values: [42] template: metadata: labels: { job: reindex } spec: restartPolicy: Never # Never | OnFailure containers: - name: reindex image: registry.example.com/search/reindex:2.4.1 args: ["--shard=$(JOB_COMPLETION_INDEX)"] resources: requests: { cpu: 500m, memory: 512Mi }
yaml
apiVersion: batch/v1 kind: CronJob metadata: name: nightly-report spec: schedule: "0 3 * * *" # minute hour day-of-month month day-of-week timeZone: "Asia/Seoul" # stable since 1.27; default is UTC concurrencyPolicy: Forbid # Allow | Forbid | Replace startingDeadlineSeconds: 300 # skip if it could not start within 5 min suspend: false successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 1 jobTemplate: spec: backoffLimit: 2 ttlSecondsAfterFinished: 86400 template: spec: restartPolicy: OnFailure containers: - name: report image: registry.example.com/reports/nightly:1.2.0
bash
kubectl create job --from=cronjob/nightly-report nightly-report-manual kubectl wait --for=condition=complete job/reindex-20261115 --timeout=1800s kubectl logs job/reindex-20261115 --all-containers kubectl get pods -l job-name=reindex-20261115 # one pod per attempt with Never

Key Comparison Table

SettingrestartPolicy: NeverrestartPolicy: OnFailure
Retry locationNew pod per failureContainer restart in place
backoffLimit countsFailed podsContainer restarts
Failed pod left behindYes (logs preserved)No
Best forLong batch jobs, debuggingCheap idempotent jobs
podFailurePolicyApplies to pod deletion/failureApplies, matches container exit
CronJob fieldDefaultEffect
concurrencyPolicyAllowForbid skips a tick if the last run is active; Replace kills it
startingDeadlineSecondsunsetA run that cannot start in time is counted as missed
successfulJobsHistoryLimit3Completed Jobs kept
failedJobsHistoryLimit1Failed Jobs kept
suspendfalseStops new runs without deleting the CronJob
timeZoneUTC (node local before 1.27)Schedule interpretation
SymptomMeaning
BackoffLimitExceededRetries exhausted; read the failed pod's log
DeadlineExceededactiveDeadlineSeconds reached
Job Running but no podsQuota or unschedulable pod
CronJob ran twice at onceconcurrencyPolicy: Allow with a slow job
CronJob never firesInvalid schedule, suspended, or past missed-run threshold

Practical Edge Cases & Deep Dives

1. completions, parallelism, and the work queue pattern

With completions: 4, parallelism: 2, the controller runs two pods, and as each succeeds it starts another until four successes accumulate. That is the fixed-amount pattern. The work-queue pattern drops completions entirely and sets only parallelism: the Job succeeds when at least one pod exits 0 and no pods are running, which suits workers that pull from a queue until it is empty.

bash
# Completions count successes, not pods: index 1 failing twice still needs 4 successes kubectl get pods -l job-name=reindex-20261115 # reindex-20261115-0-abcde 0/1 Error # reindex-20261115-0-fghij 0/1 Completed <- success 1 of 4

completionMode: Indexed gives each pod JOB_COMPLETION_INDEX from 0 to completions-1 and a stable hostname (<job>-<index> in a headless Service), which is how a Job partitions 100 inputs across 10 workers without a queue. In indexed mode completions is required, parallelism may be lower than it, and a failed index is retried as the same index.

2. What backoffLimit actually counts, and the retry ladder

Retries use exponential backoff between 10 seconds and 6 minutes (the delay doubles per failure up to the cap), and backoffLimit: 6 is the default. With restartPolicy: Never, every attempt is a new pod, so kubectl get pods shows one pod per attempt, and the failed ones remain for inspection unless ttlSecondsAfterFinished or a manual delete removes them.

bash
$ kubectl describe job reindex-20261115 | tail -5 Warning BackoffLimitExceeded Job has reached the specified backoff limit $ kubectl get job reindex-20261115 -o jsonpath='{.status.failed}{"\n"}' 4

A misconfigured backoffLimit: 0 means one failure fails the Job immediately — useful for a migration that must not retry, and a trap when a transient 429 is expected. A pod evicted or lost with its node also consumes the budget.

3. podFailurePolicy: distinguishing retryable from fatal

yaml
podFailurePolicy: rules: - action: Ignore # don't count against backoffLimit onPodConditions: - type: DisruptionTarget # eviction, preemption, drain - action: Fail # stop immediately onExitCodes: operator: In values: [2, 78] - action: FailJob # mark the whole Job failed onExitCodes: operator: In values: [137] # e.g. OOMKilled means "raise limits"

Without a policy, exit 2 (bad input) burns the same budget as exit 137 (a transient OOM), and a batch Job spends twenty minutes retrying something that can never succeed. action: Ignore with onPodConditions: DisruptionTarget is the standard fix for Jobs that keep failing a drain.

4. CronJob timing: missed runs, overlap, and timezone

text
# Controller was down 09:00-11:00 for a "*/10 * * * *" schedule # It wakes at 11:00 and evaluates the last 10 missed times. # startingDeadlineSeconds unset => it starts one run immediately and # counts the others as missed; more than 100 missed runs => no run at all.

Three behaviours matter. A run whose start time is older than startingDeadlineSeconds is skipped rather than started late. If more than 100 schedules were missed while the controller was down, the CronJob logs Cannot determine if job needs to be started and does nothing until the next valid tick. And timeZone defaults to UTC for a CronJob spec, so 0 3 * * * runs at noon KST unless you set timeZone: Asia/Seoul — the single most common "it runs at the wrong hour" cause.

concurrencyPolicy: Forbid skips a tick when the previous Job is still running; Replace kills the running Job and starts a new one, which is right for a snapshot job and wrong for anything that must finish.

5. Jobs that never finish: sidecars, probes, and exit codes

A Job pod never completes when a sidecar keeps running after the main container exits — before native sidecars (Kubernetes 1.29+), the answer was to make the sidecar exit itself. A Job pod with a failing readiness probe is not failed either; it is simply not Ready, and the Job waits. And an exit code of 143 (SIGTERM) counts as failure, so a pod killed by activeDeadlineSeconds or an eviction consumes the backoff budget.

bash
kubectl get pod -l job-name=reindex-20261115 -o jsonpath=\ '{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[0].state.terminated.exitCode}{"\n"}{end}'

Common Runtime & Configuration Mistakes

SymptomCauseFix
Job stays Running foreverSidecar never exits, or probe never passesNative sidecar, or drop/relax the probe
BackoffLimitExceeded on a permanent errorRetry budget spent on a fatal exitpodFailurePolicy with action: Fail
Job Failed after one podbackoffLimit: 0Raise it, or accept fail-fast
DeadlineExceeded mid-workWall-clock cap too tightRaise activeDeadlineSeconds; checkpoint work
Failed pods pile upNever keeps every attemptttlSecondsAfterFinished, manual cleanup
CronJob runs at the wrong hourtimeZone unset (UTC)Set timeZone explicitly
Two runs overlapconcurrencyPolicy: AllowForbid, or make the job idempotent
CronJob silently stops>100 missed schedulesFix controller availability; re-create the CronJob
Forbidden: exceded quota on Job creationNamespace quota counts Job podsRaise count/pods in the ResourceQuota

A Job is a completion contract, and every question about it reduces to three numbers: how many successes are required, how many failures are allowed, and whether anything is still running. When a batch workload misbehaves, read status.conditions, then the failed pod's exit code, then the retry budget in that order.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.