Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes Services: ClusterIP, NodePort, LoadBalancer, and Headless DNS

A Service is a label query plus a packet rewrite. Endpoints only exist for ready pods, targetPort has to match a real listener, and externalTrafficPolicy decides whether you see the client's IP.

~6 min readKubernetes
#Kubernetes#Networking#Services
3D illustration of a service virtual IP distributing traffic to pod cubes with a headless DNS fan-out

Quick Summary

A Service is a stable virtual IP plus a DNS name in front of a set of pods chosen by labels. ClusterIP is the default and is reachable only inside the cluster: kube-proxy programs iptables/IPVS rules that rewrite packets destined for the virtual IP to a backing pod. NodePort additionally opens the same port on every node. LoadBalancer asks the cloud provider (or MetalLB) for an external IP and routes it to the NodePort. ExternalName is not proxying at all — it is a CNAME returned by DNS.

The endpoints behind a Service are not the pods matched by the selector; they are the ready pods matched by the selector, materialized in EndpointSlice objects. That single fact explains most Service failures: a selector that matches nothing gives a Service with no endpoints (connection refused), a pod failing its readiness probe is removed from the endpoints (traffic skips it), and a pod without a readiness probe is assumed ready as soon as the container runs.

A headless Service (clusterIP: None) does no proxying at all: DNS returns the pod IPs directly, which is how StatefulSets give each pod a stable name (pod-0.svc.namespace.svc.cluster.local) and how clients that need per-pod connections discover them. This guide covers the four types, port mapping (port versus targetPort versus nodePort), endpoint mechanics, traffic policy, DNS naming, and the debugging sequence that finds why a connection is refused.

Basic Syntax & Core Signatures

yaml
apiVersion: v1 kind: Service metadata: name: api namespace: default spec: type: ClusterIP # ClusterIP | NodePort | LoadBalancer | ExternalName selector: app: api ports: - name: http # required when there is more than one port port: 80 # the Service's port targetPort: 8080 # the container's port (number or named port) protocol: TCP - name: metrics port: 9090 targetPort: metrics
yaml
apiVersion: v1 kind: Service metadata: name: api-node spec: type: NodePort selector: { app: api } ports: - port: 80 targetPort: 8080 nodePort: 30080 # optional; otherwise an unused port in 30000-32767
yaml
apiVersion: v1 kind: Service metadata: name: api-lb annotations: service.beta.kubernetes.io/aws-load-balancer-type: nlb # provider-specific spec: type: LoadBalancer externalTrafficPolicy: Local # preserve client IP; only nodes with a pod answer selector: { app: api } ports: - port: 443 targetPort: 8443
yaml
apiVersion: v1 kind: Service metadata: name: db spec: clusterIP: None # headless publishNotReadyAddresses: true # include pods that are not ready yet selector: { app: db } ports: - port: 5432 targetPort: 5432
bash
kubectl get svc,endpointslice -o wide kubectl get endpointslice -l kubernetes.io/service-name=api -o yaml kubectl describe svc api kubectl run net --rm -it --image=nicolaka/netshoot -- bash # getent hosts api.default.svc.cluster.local # curl -sv http://api:80/readyz # dig +short api.default.svc.cluster.local kubectl port-forward svc/api 8080:80

Key Comparison Table

TypeReachable fromWhat it createsTypical use
ClusterIPInside the clusterVirtual IP + DNSInternal service-to-service
NodePortAny node IP, port 30000-32767ClusterIP + port on every nodeBare-metal, quick external access
LoadBalancerExternal IP from the providerNodePort + provider LBPublic ingress
ExternalNameInside the clusterCNAME record, no proxyPoint at an external hostname
Headless (clusterIP: None)Inside the clusterPod IPs in DNS, no VIPStatefulSets, per-pod clients
FieldMeaningDefault
portPort the Service listens onRequired
targetPortContainer port traffic is sent toSame as port
nodePortPort exposed on each nodeAllocated from 30000-32767
protocolTCP, UDP, SCTPTCP
sessionAffinityNone or ClientIPNone
externalTrafficPolicyCluster or LocalCluster
internalTrafficPolicyCluster or LocalCluster
publishNotReadyAddressesInclude not-ready pods in DNS/endpointsfalse
SymptomMeaning
Connection refused immediatelyService has zero endpoints (selector mismatch, no ready pods)
Connection timed outA policy/firewall blocked it, or kube-proxy is not programming the VIP
DNS resolves, connection refusedEndpoints question, not DNS
DNS does not resolveWrong namespace/name, or CoreDNS problem
Works via port-forward, not via ServicetargetPort or selector problem
Works inside the cluster, not outsideClusterIP is not routable from outside
503 from an ingress, 200 from pod IPEndpoints empty or readiness failing

Practical Edge Cases & Deep Dives

1. A Service with no endpoints: the selector is the suspect

bash
$ kubectl get endpointslices -l kubernetes.io/service-name=api NAME ADDRESSTYPE PORTS ENDPOINTS AGE api-abc12 IPv4 8080 <unset> 14m $ kubectl get svc api -o jsonpath='{.spec.selector}' {"app":"api"} $ kubectl get pods -l app=api --show-labels NAME READY STATUS LABELS api-1 0/1 Running app=api,tier=web # not ready

An EndpointSlice with <unset> endpoints and a Service that matches pods means the pods are not ready — readiness probe failing, Ready=False, or terminating. kubectl get pods -l with the Service's selector reproduces exactly what the endpoints controller sees. A selector that matches nothing (a typo, or labels on the Deployment's template not on the pods) produces the same empty slice, and the failure is indistinguishable from outside: the connection is refused because there is nothing to rewrite the packet to.

bash
kubectl get pods -l "$(kubectl get svc api -o jsonpath='{.spec.selector}' | tr -d '{}"' | tr ',' ',')"

2. port versus targetPort versus the pod's actual listener

yaml
ports: - port: 80 # clients connect to api:80 targetPort: 8080 # and the packet is DNAT'd to pod-IP:8080

A common misconfiguration is a container that listens on 127.0.0.1:8080 instead of 0.0.0.0:8080. The pod's own IP is a different interface, so traffic forwarded to pod-IP:8080 is refused even though curl localhost:8080 inside the container works. The other is a named targetPort that does not match any containerPort name, in which case the EndpointSlice has the wrong port or none.

bash
kubectl exec deploy/api -- ss -tlnp # confirms 0.0.0.0 vs 127.0.0.1 kubectl get endpointslice -l kubernetes.io/service-name=api -o jsonpath='{.items[*].ports}'

3. NodePort, LoadBalancer, and where the client IP goes

bash
$ curl -s https://api.example.com/whoami {"remote":"10.244.3.7"} # externalTrafficPolicy: Cluster — the node's IP $ kubectl patch svc api-lb -p '{"spec":{"externalTrafficPolicy":"Local"}}' $ curl -s https://api.example.com/whoami {"remote":"203.0.113.42"} # the real client IP

With Cluster (the default), a request that lands on a node without a local pod is forwarded again, and the source address becomes the node's, which breaks IP allowlists and rate limiting. With Local, kube-proxy only routes to local pods, so client IP survives — but nodes without a pod fail health checks and get removed from the load balancer, which is the correct behaviour and also the reason to run enough replicas or a DaemonSet. NodePort on a cloud load balancer exposes the node port publicly unless a firewall or security group blocks it, which is a frequent accidental exposure.

4. Headless services, StatefulSets, and DNS records

bash
$ kubectl run net --rm -it --image=nicolaka/netshoot -- dig +short db.default.svc.cluster.local 10.244.1.5 10.244.2.8 10.244.3.2 $ dig +short -x 10.244.2.8 db-1.db.default.svc.cluster.local. $ dig +short SRV _5432._tcp.db.default.svc.cluster.local 0 33 5432 db-1.db.default.svc.cluster.local.

A headless Service returns one A record per ready pod, and for a StatefulSet each pod gets a stable hostname pod-N.service.namespace.svc.cluster.local that survives rescheduling. publishNotReadyAddresses: true is required for peer discovery at bootstrap, when pods are intentionally not ready yet — without it, a cluster that needs to talk to itself to become ready deadlocks. Headless DNS returns all addresses and clients choose; there is no load balancing, no sessionAffinity, and no virtual IP to debug.

5. DNS search paths and the name that resolves to the wrong namespace

bash
$ cat /etc/resolv.conf search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5 $ getent hosts api # api.default.svc.cluster.local if in default $ getent hosts api.prod # api.prod.svc.cluster.local — the namespace part matters

A bare api resolves within the client's own namespace first. Calling a Service in another namespace needs api.prod or the FQDN api.prod.svc.cluster.local, and a name that exists in both namespaces silently resolves to the wrong one. ndots:5 means any name with fewer than five dots is tried against every search suffix before the absolute name, which makes external hostnames slower to resolve and, in rare cases, resolves an internal name you did not intend.

6. kube-proxy modes and the "works on two of three pods" pattern

bash
kubectl -n kube-system get pods -l k8s-app=kube-proxy -o wide kubectl -n kube-system logs ds/kube-proxy --tail=50 | grep -i -E 'error|sync' kubectl get svc api -o jsonpath='{.spec.clusterIP}{"\n"}'

In iptables mode, each connection is DNAT'd to a randomly chosen endpoint at connect time; long-lived connections therefore stick to one pod, and a pod that dies mid-connection produces resets until the client reconnects. In IPVS mode, conntrack and schedulers (rr, lc, sh) behave differently and the ipvsadm -Ln output on a node is authoritative. Neither mode retries a broken connection, so "one third of requests fail" usually means one endpoint in three is unhealthy but still in the endpoints list — which points back to the readiness probe.

Common Runtime & Configuration Mistakes

SymptomCauseFix
Connection refusedService has no endpointsCheck selector and pod readiness
Works via port-forward, not via ServicetargetPort mismatchAlign targetPort with the actual listener
curl localhost works in the pod, Service failsApp bound to 127.0.0.1Bind 0.0.0.0
Same Service name resolves across namespacesShort name resolves to the client's namespaceUse svc.namespace or the FQDN
Client IP is a node addressexternalTrafficPolicy: ClusterSet Local and size replicas accordingly
Load balancer marks nodes unhealthyLocal with no pod on every nodeMore replicas or a DaemonSet
Multiple ports rejectedPorts not namedAdd name: to every entry
nodePort out of rangeOutside 30000-32767Omit it and let the allocator choose
Endpoints include a terminating podReadiness still true during terminationAdd a preStop drain; readiness should fail fast
StatefulSet peers never connect at bootHeadless without publishNotReadyAddressesSet it true for that Service
ExternalName fails from outside the clusterNo DNS record is created, only a CNAMEResolve it inside, or use a real Service
Changing clusterIP is rejectedImmutable fieldDelete and recreate the Service

Services are a label query plus a packet rewrite, and every failure comes back to one of those two halves: does the selector match ready pods, and does the port arithmetic land on something that is actually listening on the pod's IP? Checking endpointslice first answers the first question, and ss -tlnp inside the container answers the second, before any network theory is needed.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.