Amazon SQS for Data Engineers

Standard vs FIFO queues, visibility timeout and long polling, boto3 producers and consumers, DLQs and redrive, Lambda integration and pipeline patterns.

1. What Amazon SQS Is

Amazon Simple Queue Service is a fully managed message queue. Producers send messages; consumers receive, process and delete them. Queues decouple components, absorb bursts, and let work be retried safely.

2. Producing & Consuming with boto3

import json, boto3

sqs = boto3.client("sqs")
QUEUE_URL = sqs.get_queue_url(QueueName="file-jobs")["QueueUrl"]

# send in batches of up to 10
entries = [{"Id": str(i), "MessageBody": json.dumps({"key": k})} for i, k in enumerate(keys[:10])]
sqs.send_message_batch(QueueUrl=QUEUE_URL, Entries=entries)

# worker loop: long poll, process, delete only on success
while True:
    resp = sqs.receive_message(QueueUrl=QUEUE_URL, MaxNumberOfMessages=10,
                               WaitTimeSeconds=20, VisibilityTimeout=300)
    for msg in resp.get("Messages", []):
        try:
            process(json.loads(msg["Body"]))  # must be idempotent
            sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])
        except Exception as exc:
            print("will retry after visibility timeout:", exc)  # not deleted → redelivered

# FIFO: ordering per account, deduplication per transaction
sqs.send_message(
    QueueUrl=FIFO_URL,
    MessageBody=json.dumps(txn),
    MessageGroupId=txn["account_id"],  # ordered within an account; groups run in parallel
    MessageDeduplicationId=txn["txn_id"],  # duplicates within 5 minutes are dropped
)

3. Dead-Letter Queues & Redrive

aws sqs create-queue --queue-name file-jobs-dlq
aws sqs set-queue-attributes --queue-url https://sqs.ap-south-1.amazonaws.com/111122223333/file-jobs \
  --attributes '{"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-south-1:111122223333:file-jobs-dlq\",\"maxReceiveCount\":\"5\"}"}'

# after fixing the bug, move messages back to the source queue
aws sqs start-message-move-task --source-arn arn:aws:sqs:ap-south-1:111122223333:file-jobs-dlq

Always alarm on DLQ depth (ApproximateNumberOfMessagesVisible > 0) so failures are noticed, not silently parked.

4. SQS with Lambda

• Lambda polls the queue and invokes your function with batches (up to 10,000 with a batching window).
• Return batchItemFailures (partial batch response) so only failed messages are retried; see the Lambda guide.
• Set maximum concurrency on the event source to protect downstream databases.
• Make the queue's visibility timeout at least 6× the function timeout.

5. Data Engineering Patterns

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowS3ToSend",
    "Effect": "Allow",
    "Principal": { "Service": "s3.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:ap-south-1:111122223333:file-jobs",
    "Condition": { "ArnEquals": { "aws:SourceArn": "arn:aws:s3:::acme-lake-prod" } }
  }]
}

This queue policy lets S3 event notifications from one bucket send to the queue. Enable server-side encryption (SSE-SQS or SSE-KMS); with KMS, the sending service also needs permission on the key.

6. Common Interview Questions