Amazon MSK for Data Engineers

Managed Kafka options, bootstrap brokers and authentication, IAM producers and Spark consumers, MSK Connect, sizing and operations.

1. What Amazon MSK Is

Amazon Managed Streaming for Apache Kafka (MSK) runs Apache Kafka for you: brokers, storage, patching and recovery across availability zones. Your producers, consumers and tools use the standard Kafka APIs, so everything in the Kafka guide applies.

2. Connecting & Authentication

aws kafka list-clusters-v2 --query "ClusterInfoList[].{name:ClusterName, arn:ClusterArn}"
aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:ap-south-1:111122223333:cluster/txn-cluster/abc-123

# Python producer with IAM auth: pip install confluent-kafka aws-msk-iam-sasl-signer-python
import json
from confluent_kafka import Producer
from aws_msk_iam_sasl_signer import MSKAuthTokenProvider

REGION = "ap-south-1"

def oauth_cb(oauth_config):
    token, expiry_ms = MSKAuthTokenProvider.generate_auth_token(REGION)
    return token, expiry_ms / 1000                         # seconds

producer = Producer({
    "bootstrap.servers": "b-1.txncluster.abc123.c2.kafka.ap-south-1.amazonaws.com:9098",
    "security.protocol": "SASL_SSL",
    "sasl.mechanisms": "OAUTHBEARER",
    "oauth_cb": oauth_cb,
    "acks": "all",
    "enable.idempotence": True,
})
producer.produce("txn.events", key="ACC1", value=json.dumps({"txn_id": "T1", "amount": 250}))
producer.flush()

# Spark Structured Streaming reading MSK with IAM (needs the aws-msk-iam-auth JAR on the classpath)
stream = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", BOOTSTRAP)
    .option("subscribe", "txn.events")
    .option("kafka.security.protocol", "SASL_SSL")
    .option("kafka.sasl.mechanism", "AWS_MSK_IAM")
    .option("kafka.sasl.jaas.config", "software.amazon.msk.auth.iam.IAMLoginModule required;")
    .option("kafka.sasl.client.callback.handler.class",
            "software.amazon.msk.auth.iam.IAMClientCallbackHandler")
    .option("startingOffsets", "latest")
    .load())

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["kafka-cluster:Connect", "kafka-cluster:DescribeCluster"],
      "Resource": "arn:aws:kafka:ap-south-1:111122223333:cluster/txn-cluster/*" },
    { "Effect": "Allow", "Action": ["kafka-cluster:ReadData", "kafka-cluster:DescribeTopic"],
      "Resource": "arn:aws:kafka:ap-south-1:111122223333:topic/txn-cluster/*/txn.events" },
    { "Effect": "Allow", "Action": ["kafka-cluster:AlterGroup", "kafka-cluster:DescribeGroup"],
      "Resource": "arn:aws:kafka:ap-south-1:111122223333:group/txn-cluster/*/fraud-scoring" }
  ]
}

3. MSK Connect

MSK Connect runs Kafka Connect connectors (packaged as custom plugins from S3) with autoscaling workers. Typical data-engineering connectors:

4. Sizing & Operations

• Spread brokers across three AZs; replication factor 3 with min.insync.replicas = 2 and producers with acks=all survive an AZ failure without data loss.
• Partitions set consumer parallelism; plan for peak throughput per partition and avoid too many partitions per broker.
• Tiered storage keeps older segments in cheaper storage, enabling long retention for Kappa-style replays.
• Monitor: MaxOffsetLag / SumOffsetLag (consumer lag), CpuUser, KafkaDataLogsDiskUsed, under-replicated partitions.
• Use Glue Schema Registry (or Confluent Schema Registry) for Avro/Protobuf contracts.

5. Common Interview Questions