DevOps Reference

CI/CD pipelines, Docker, Terraform, release strategies, testing data pipelines, observability, DataOps and MLOps, and security in the pipeline.

1. What DevOps Is

DevOps is a set of cultural practices and tooling that brings development and operations together so teams can ship changes frequently, safely and reliably. The core ideas: automate everything repeatable, keep everything in version control, get fast feedback, and own what you build in production.

Exact DORA thresholds shift between yearly reports; the point is to measure speed and stability together.

2. The CI/CD Pipeline

 commit ─► CI: build ─► lint ─► unit tests ─► package artifact (wheel / image) ─► security scans
                                                    │
 CD:  deploy to dev ─► integration + data-quality tests ─► deploy to test/preprod ─► approval ─► prod
                                                    │
                       monitor ─► alert ─► roll back or roll forward

A GitHub Actions workflow running tests on every pull request:

name: ci
on:
  pull_request:
  push:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - uses: actions/setup-java@v4          # PySpark needs a JVM
        with: { distribution: temurin, java-version: "17" }
      - run: pip install -r requirements-dev.txt
      - run: ruff check src tests
      - run: pytest -q

3. Containers with Docker

A container image packages code, runtime and dependencies so the job runs the same on a laptop, in CI and in production.

# Dockerfile for a Python data job
FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # cached layer: only rebuilt when requirements change

COPY src/ ./src/
RUN useradd --create-home appuser
USER appuser                                         # don't run as root

ENTRYPOINT ["python", "-m", "src.jobs.export"]

docker build -t myregistry/exporter:2.0.1 .
docker run --rm --env-file .env myregistry/exporter:2.0.1 --lookback-days 1
docker push myregistry/exporter:2.0.1
docker images
docker ps -a
docker logs -f <container>
docker exec -it <container> sh
docker compose up -d         # multi-container local stacks (Kafka + Spark + Mongo)

• Layers are cached top-down: copy dependency files and install them before copying source code, so code edits do not reinstall everything.
• Multi-stage builds compile or build in one stage and copy only the result into a slim final image.
• Tag images with immutable versions or Git SHAs, never rely on :latest in deployments, and scan images for vulnerabilities (Trivy, ECR/ACR scanning).
• Images are run on Kubernetes, ECS/Fargate, AWS Batch, Cloud Run, or as Databricks container services.

4. Infrastructure as Code with Terraform

Infrastructure as Code defines cloud resources in version-controlled files, so environments are reproducible, reviewed in pull requests, and rebuilt on demand. Terraform is the most common tool; CloudFormation, Bicep and Pulumi are alternatives.

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
  backend "s3" {                       # shared, locked state
    bucket       = "acme-terraform-state"
    key          = "data-platform/prod.tfstate"
    region       = "ap-south-1"
    use_lockfile = true                # S3-native locking (older setups use a DynamoDB table)
  }
}

provider "aws" { region = var.region }

resource "aws_s3_bucket" "raw" {
  bucket = "acme-${var.env}-raw"
  tags   = { env = var.env, owner = "data-platform" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "raw" {
  bucket = aws_s3_bucket.raw.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.data.arn
    }
  }
}

terraform init       # download providers, configure the backend
terraform fmt        # format files
terraform validate
terraform plan -var-file=prod.tfvars -out=plan.tfplan    # preview changes
terraform apply plan.tfplan                              # apply exactly what was planned
terraform destroy    # tear down (careful)

• State maps code to real resources. Keep it in a remote backend with locking and encryption, never in Git, and never edit it by hand.
• Use modules for reusable building blocks (a standard bucket, a Databricks workspace) and separate state per environment.
• Run plan in CI on every pull request and apply only from the pipeline after approval. Watch for drift: changes made by hand in the console.

5. Environments & Release Strategies

Data pipelines have their own version of blue/green: write the new version's output to new tables (or a shadow run), reconcile against the old output, then switch consumers. Build once and promote the same artifact through dev → test → prod, changing only configuration.

6. Testing Data Pipelines

# tests/test_transform.py
import pytest
from pyspark.sql import SparkSession
from src.transform import dedupe_latest

@pytest.fixture(scope="session")
def spark():
    return (SparkSession.builder.master("local[2]")
            .config("spark.sql.shuffle.partitions", "2")    # fast tests
            .getOrCreate())

def test_dedupe_keeps_latest_row(spark):
    df = spark.createDataFrame(
        [(1, "old", "2026-01-01"), (1, "new", "2026-01-02"), (2, "only", "2026-01-01")],
        ["id", "val", "updated_at"])
    out = {r.id: r.val for r in dedupe_latest(df).collect()}
    assert out == {1: "new", 2: "only"}

7. Monitoring & Observability

• Define SLIs (what you measure, e.g. "gold table ready by 06:00"), SLOs (the target, e.g. 99% of days) and alert when the error budget is burning, not on every blip.
• For pipelines, alert on: job failure, duration well above normal, missing or late input, row-count anomalies, DQ rule failures, and consumer lag for streams.
• Every alert needs an owner and a runbook. Write blameless post-mortems after incidents.

8. DataOps & MLOps

DataOps applies DevOps to data: version-controlled transformations, automated testing of code and data, CI/CD for pipelines, and monitoring of data quality in production.

9. Security in the Pipeline (DevSecOps)

• Secrets: in a vault (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault) or the CI secret store; never in code, images or logs.
• Keyless CI auth: OIDC federation from GitHub Actions, GitLab or Jenkins to AWS/Azure/GCP roles instead of long-lived access keys.
• Scanning: SAST (CodeQL, Semgrep), dependency scanning (Dependabot, pip-audit), container image scanning (Trivy), IaC scanning (Checkov, tfsec), secret scanning (gitleaks).
• Least privilege: CI roles that can deploy only what they need; production changes only through the pipeline; humans read-only in prod.
• Supply chain: pin dependency versions, use trusted base images, sign images (cosign) and keep an SBOM.

10. Tool Landscape

11. Common Interview Questions