TL;DR: Kubernetes availability comes from application behavior plus platform configuration. Define readiness, liveness, startup, resources, graceful shutdown, disruption budgets, rollout limits, observability, and recovery procedures before adding more replicas.
Reliability Starts with a Contract
Before writing manifests, define the service’s availability target, acceptable latency, dependency behavior, recovery time, and data-loss tolerance. Kubernetes can restart a process, but it cannot decide whether a partially initialized application is safe to receive traffic.
Use Probes for Different Questions
A startup probe answers whether the application has finished initialization. A readiness probe answers whether it should receive traffic now. A liveness probe answers whether the process is stuck and should be restarted.
Do not make liveness depend on a database query unless a database outage should restart every replica. A dependency can be unavailable while the process remains healthy enough to serve cached or degraded responses. Probe design should reflect the service contract.
Set Resource Requests and Limits
Requests influence scheduling; limits constrain consumption. Measure real CPU and memory usage under normal and peak load before choosing values. A low memory limit can create restart loops, while an oversized request can prevent useful scheduling.
Observe throttling, out-of-memory kills, garbage collection, queue depth, and response latency. Resource configuration is a hypothesis that should be revised from production evidence.
Design Graceful Shutdown
On termination, stop accepting new work, fail readiness, allow in-flight requests to complete, close connections, and exit before the termination grace period ends. Long-running jobs need cancellation or checkpointing so a rollout does not silently lose work.
Test termination during active requests and during queue processing. A clean SIGTERM handler is more valuable than a larger replica count.
Make Rollouts Reversible
Use rolling updates with a maximum unavailable budget and a maximum surge appropriate to the service. Keep the previous image available, deploy small changes, and define an automated or operator-triggered rollback path.
Readiness must represent the new version’s real ability to serve traffic. A deployment can report “available” while returning errors if the probe only checks that a port is open.
Protect Against Voluntary Disruption
Pod disruption budgets protect availability during voluntary events such as node drains. They do not protect against every failure and should not be so strict that cluster maintenance becomes impossible. Combine them with replica distribution across failure domains.
Observability Before Autoscaling
Collect logs, metrics, traces, deployment events, probe failures, restart counts, saturation, and dependency latency. Autoscaling on CPU alone is often wrong for queue workers or I/O-bound APIs. Scale on the signal that represents user impact or backlog.
Run Failure Drills
Delete a pod during traffic, make a dependency slow, exhaust a resource limit in staging, roll back a bad image, and drain a node. Record the observed behavior, time to recovery, and missing alert. Reliability is learned through controlled evidence.
Production Checklist
- Availability and recovery objectives are documented.
- Startup, readiness, and liveness probes answer different questions.
- Requests and limits are based on measurements.
- Shutdown drains work safely.
- Rollouts are small, observable, and reversible.
- Replicas are distributed across failure domains.
- Alerts identify user impact, not only pod state.
- Recovery drills are repeated after major architecture changes.
Example Deployment Shape
A production deployment should make its assumptions visible. Define a small number of replicas, realistic resource requests, separate readiness and liveness endpoints, a termination grace period, and a rolling-update strategy. Keep configuration and secrets outside the image, and make the image immutable so the same artifact can move through environments.
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
spec:
terminationGracePeriodSeconds: 45
containers:
- name: api
image: example/api:2026-08-14
readinessProbe:
httpGet: { path: /ready, port: 3000 }
livenessProbe:
httpGet: { path: /live, port: 3000 }
resources:
requests: { cpu: 100m, memory: 256Mi }
limits: { cpu: 500m, memory: 512Mi }
The values are examples, not universal defaults. Measure startup time, memory behavior, and request concurrency before tuning them for your service.
Dependency Failure and Degraded Modes
Decide what the service should do when a dependency is slow or unavailable. It may return a cached response, queue work, disable a non-critical feature, or fail quickly with a useful error. A readiness probe should not automatically restart every pod because one downstream service is unavailable.
Set timeouts on outbound calls, bound retries, and use exponential backoff with jitter. Unlimited retries turn a partial outage into a traffic storm. Record dependency latency and error budgets so the operator can see the real failure chain.
Capacity and Scheduling
Replica count alone does not create capacity. Confirm that nodes, pod quotas, IP ranges, storage, and downstream databases can handle the desired scale. Use topology spread constraints where availability zones matter, and reserve capacity for system workloads.
Load-test with realistic payloads and concurrency. Watch queue depth, CPU throttling, memory pressure, database connections, and p95 latency together. Scaling the API while the database is saturated only moves the bottleneck.
Incident Runbook
For each critical service, document how to identify a bad rollout, pause or roll back a deployment, inspect probe failures, drain a node safely, and restore a dependency. Include commands, dashboards, owners, and escalation paths. Run the procedure during business hours before a real incident forces someone to invent it.
Production Checklist
- Availability and recovery objectives are documented.
- Startup, readiness, and liveness probes answer different questions.
- Requests and limits are based on measurements.
- Shutdown drains work safely.
- Rollouts are small, observable, and reversible.
- Replicas are distributed across failure domains.
- Dependency timeouts and bounded retries are configured.
- Alerts identify user impact, not only pod state.
- Recovery drills and runbooks are tested.
For teams moving from a single server to reliable cloud infrastructure, our DevOps and cloud deployment services cover deployment design, observability, and safe scaling.