Kubernetes Reference

Cluster architecture, core objects, manifests and kubectl, scaling and health checks, batch jobs, Spark on Kubernetes, troubleshooting, and Helm.

1. What Kubernetes Is

Kubernetes (K8s) runs containers across a cluster of machines. You declare the desired state ("run 3 copies of this image, reachable on port 80") and controllers continuously work to make the actual state match: restarting crashed containers, rescheduling them when a node dies, and scaling them up or down.

            ┌──────────────── CONTROL PLANE ────────────────┐
 kubectl ──►│ API server ◄──► etcd (cluster state)          │
            │ scheduler (picks a node for each new pod)     │
            │ controller manager (reconciles desired state) │
            └───────────────────────┬───────────────────────┘
                 ┌──────────────────┼──────────────────┐
              NODE 1             NODE 2             NODE 3
          kubelet · kube-proxy · container runtime (containerd)
          [pod] [pod]          [pod] [pod]         [pod]

2. Core Objects

3. A Deployment and Service

Manifests are YAML files applied with kubectl apply -f. This one runs three replicas of an API and exposes them inside the cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-api
  namespace: data
spec:
  replicas: 3
  selector:
    matchLabels: { app: fraud-api }
  template:
    metadata:
      labels: { app: fraud-api }
    spec:
      containers:
        - name: api
          image: myregistry/fraud-api:1.4.2      # pin a version, never :latest
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef: { name: fraud-api-config }
          resources:
            requests: { cpu: "250m", memory: "512Mi" }
            limits:   { memory: "1Gi" }
          readinessProbe:
            httpGet: { path: /health, port: 8080 }
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: fraud-api
  namespace: data
spec:
  selector: { app: fraud-api }       # routes to pods with this label
  ports:
    - port: 80
      targetPort: 8080

Other pods reach it at http://fraud-api.data.svc.cluster.local (or just fraud-api from the same namespace). Labels and selectors are how Deployments find their pods and Services find their endpoints.

4. kubectl Cheat Sheet

kubectl config get-contexts / use-context prod      # which cluster am I talking to?
kubectl get pods -n data -o wide                    # pods with node and IP
kubectl get deploy,svc,ingress -n data
kubectl describe pod fraud-api-7d9f -n data         # events: scheduling, pulls, probe failures
kubectl logs fraud-api-7d9f -n data -f              # follow logs (add -c <container>, --previous)
kubectl exec -it fraud-api-7d9f -n data -- sh       # shell inside the container
kubectl apply -f k8s/                               # create or update from manifests
kubectl diff -f k8s/                                # preview changes before applying
kubectl rollout status deploy/fraud-api -n data
kubectl rollout undo deploy/fraud-api -n data       # roll back to the previous version
kubectl scale deploy/fraud-api --replicas=5 -n data
kubectl port-forward svc/fraud-api 8080:80 -n data  # test a service from your laptop
kubectl top pods -n data                            # CPU / memory (needs metrics-server)
kubectl get events -n data --sort-by=.lastTimestamp

5. EKS: Checking Pods & Logs

1. Connect kubectl to the EKS cluster

aws sts get-caller-identity                          # which IAM user/role am I?
aws eks list-clusters --region ap-south-1
aws eks update-kubeconfig --region ap-south-1 --name data-platform-prod --alias prod
kubectl config current-context                       # confirm you are on the right cluster
kubectl get nodes -o wide                            # nodes, status, versions, IPs

If kubectl says "You must be logged in to the server (Unauthorized)", your IAM identity has no access to the cluster. An admin must add it as an access entry (aws eks list-access-entries --cluster-name data-platform-prod) or, on older clusters, to the aws-auth ConfigMap.

2. Check pods

kubectl get pods -A                                   # all namespaces
kubectl get pods -n data -o wide                      # + node and pod IP
kubectl get pods -n data -l app=fraud-api             # filter by label
kubectl get pods -n data --field-selector=status.phase!=Running   # anything not running
kubectl get pods -n data -w                           # watch status changes live
kubectl get pods -n data --sort-by='.status.containerStatuses[0].restartCount'
kubectl describe pod fraud-api-7d9f8-x2k4p -n data    # events at the bottom explain most failures
kubectl get pod fraud-api-7d9f8-x2k4p -n data -o yaml # full spec and status
kubectl get events -n data --sort-by=.lastTimestamp
kubectl top pods -n data --containers                 # CPU/memory (needs the metrics-server add-on)
kubectl top nodes

3. Read pod logs

kubectl logs fraud-api-7d9f8-x2k4p -n data                    # current logs
kubectl logs fraud-api-7d9f8-x2k4p -n data -f                 # follow (stream) new lines
kubectl logs fraud-api-7d9f8-x2k4p -n data --tail=200 --timestamps
kubectl logs fraud-api-7d9f8-x2k4p -n data --since=1h         # or --since-time=2026-10-25T02:00:00Z
kubectl logs fraud-api-7d9f8-x2k4p -n data --previous         # the crashed container before a restart
kubectl logs fraud-api-7d9f8-x2k4p -n data -c sidecar         # one container in a multi-container pod
kubectl logs fraud-api-7d9f8-x2k4p -n data --all-containers
kubectl logs fraud-api-7d9f8-x2k4p -n data -c init-db         # init container logs

kubectl logs deploy/fraud-api -n data                         # logs from one pod of a deployment
kubectl logs job/nightly-export -n data                       # logs of a Job's pod
kubectl logs -l app=fraud-api -n data --prefix --tail=100     # all matching pods, lines prefixed with pod name

kubectl logs fraud-api-7d9f8-x2k4p -n data > fraud-api.log    # save to a file

• With a label selector (-l), kubectl shows only the last 10 lines per pod unless you pass --tail, and follows at most 5 pods at once (--max-log-requests).
• --previous is the key flag for CrashLoopBackOff: the current container has just restarted and has no useful logs yet.
• For many pods at once, the open-source stern tool tails them with colour per pod: stern fraud-api -n data --since 15m.

4. Get inside a pod

kubectl exec -it fraud-api-7d9f8-x2k4p -n data -- sh                  # shell (bash if the image has it)
kubectl exec -it fraud-api-7d9f8-x2k4p -n data -c api -- env          # check environment variables
kubectl exec fraud-api-7d9f8-x2k4p -n data -- cat /app/logs/app.log   # app logs written to a file
kubectl debug -it fraud-api-7d9f8-x2k4p -n data --image=busybox --target=api   # debug container for minimal images
kubectl port-forward pod/fraud-api-7d9f8-x2k4p 8080:8080 -n data       # call the pod from your laptop

5. Spark jobs on EKS

kubectl get pods -n spark-jobs -l spark-role=driver               # driver pods
kubectl logs -f <driver-pod> -n spark-jobs                         # job progress and errors
kubectl get pods -n spark-jobs -l spark-role=executor
kubectl logs <executor-pod> -n spark-jobs --previous               # an executor that was OOMKilled
kubectl port-forward <driver-pod> 4040:4040 -n spark-jobs          # open the Spark UI at localhost:4040

Executor pods are deleted when the job ends, and their logs go with them. Set spark.kubernetes.executor.deleteOnTermination=false while debugging, and ship logs to CloudWatch or S3 for production.

6. Logs after the pod is gone: CloudWatch

kubectl only reads logs from pods that still exist. For history, install the Amazon CloudWatch Observability EKS add-on (Container Insights with Fluent Bit), which ships container logs to CloudWatch Logs:

aws logs tail /aws/containerinsights/data-platform-prod/application --follow
aws logs tail /aws/containerinsights/data-platform-prod/application \
    --since 2h --filter-pattern "ERROR"

# control-plane logs (API server, authenticator, audit) must be enabled first
aws eks update-cluster-config --region ap-south-1 --name data-platform-prod \
    --logging '{"clusterLogging":[{"types":["api","audit","authenticator"],"enabled":true}]}'
aws logs tail /aws/eks/data-platform-prod/cluster --since 30m

6. Configuration & Secrets

apiVersion: v1
kind: ConfigMap
metadata: { name: fraud-api-config, namespace: data }
data:
  KAFKA_BOOTSTRAP: "kafka:9092"
  LOG_LEVEL: "INFO"
---
apiVersion: v1
kind: Secret
metadata: { name: mongo-credentials, namespace: data }
type: Opaque
stringData:                     # plain text here; stored base64-encoded
  MONGO_USER: app
  MONGO_PASSWORD: change-me

• Inject with envFrom / env.valueFrom.secretKeyRef, or mount as files (files update when the object changes; environment variables need a pod restart).
• Secrets are only base64-encoded, not encrypted. Enable encryption at rest for etcd, restrict access with RBAC, and keep real secrets out of Git: use External Secrets Operator or the Secrets Store CSI driver with AWS Secrets Manager, Azure Key Vault or Vault.

7. Resources, Scaling & Health Checks

Pods whose requests equal their limits get the Guaranteed QoS class and are evicted last under node pressure; pods with no requests are BestEffort and evicted first.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: fraud-api, namespace: data }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: fraud-api }
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }   # % of the CPU request

The HPA adds pods; the Cluster Autoscaler or Karpenter adds nodes when pods cannot be scheduled. For Kafka consumers, KEDA can scale on consumer lag instead of CPU.

8. Jobs & CronJobs for Data Work

apiVersion: batch/v1
kind: CronJob
metadata: { name: nightly-export, namespace: data }
spec:
  schedule: "0 2 * * *"            # 02:00 daily, in the controller's time zone
  timeZone: "Asia/Kolkata"         # supported in Kubernetes 1.27+
  concurrencyPolicy: Forbid        # don't start a run while the previous one is still going
  jobTemplate:
    spec:
      backoffLimit: 2              # retries before the Job is marked failed
      activeDeadlineSeconds: 7200
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: export
              image: myregistry/exporter:2.0.1
              args: ["--lookback-days", "1"]
              envFrom:
                - secretRef: { name: warehouse-credentials }

Make jobs idempotent: a pod can be retried, or (rarely) run twice. For multi-step pipelines with dependencies, use an orchestrator (Airflow's KubernetesPodOperator, Argo Workflows) rather than chaining CronJobs by time.

9. Spark on Kubernetes

Kubernetes can replace YARN as Spark's cluster manager. spark-submit creates a driver pod, and the driver asks the API server for executor pods, deleting them when the job finishes.

spark-submit \
  --master k8s://https://<api-server>:443 \
  --deploy-mode cluster \
  --name txn-enrichment \
  --conf spark.kubernetes.namespace=spark-jobs \
  --conf spark.kubernetes.container.image=myregistry/spark-py:3.5.1 \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.executor.instances=10 \
  --conf spark.executor.cores=4 \
  --conf spark.executor.memory=12g \
  --conf spark.dynamicAllocation.enabled=true \
  --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
  local:///opt/app/jobs/enrich_txn.py --run-date 2026-10-25

• local:// means the file is already inside the image. Bake code and dependencies into the image for reproducible runs.
• The driver needs a service account allowed to create, watch and delete pods in the namespace.
• There is no external shuffle service on Kubernetes, so dynamic allocation needs shuffleTracking (or Spark 3.1+ decommissioning).
• The Spark Operator lets you declare a SparkApplication YAML instead of running spark-submit, which fits GitOps and Airflow well. Managed options include EMR on EKS.

10. Troubleshooting

11. Helm & Managed Kubernetes

Helm packages manifests into versioned, parameterised charts. One chart plus a values file per environment replaces copies of near-identical YAML:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kafka bitnami/kafka -n streaming -f values-dev.yaml
helm upgrade --install fraud-api ./charts/fraud-api -f values-prod.yaml --set image.tag=1.4.3
helm rollback fraud-api 3
helm list -n data

In a chart template, values are referenced with Go templating, e.g. image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}".

Common GitOps setup: manifests or Helm values live in Git, and Argo CD or Flux continuously syncs the cluster to match, so Git is the source of truth and every change is reviewed.