Amazon SNS & EventBridge

Pub/sub fan-out with filter policies, pipeline alerting, EventBridge rules, Scheduler, Pipes and archives, and a comparison of AWS messaging services.

1. What Amazon SNS Is

Amazon Simple Notification Service is publish/subscribe messaging: publishers send to a topic, and SNS pushes a copy to every subscription (SQS queues, Lambda functions, HTTP endpoints, email, SMS, Firehose). In data platforms it is used for fan-out and alerts.

2. Fan-Out with Filtering

TOPIC=$(aws sns create-topic --name file-arrivals --query TopicArn --output text)
aws sns subscribe --topic-arn $TOPIC --protocol sqs \
  --notification-endpoint arn:aws:sqs:ap-south-1:111122223333:loader-queue \
  --attributes '{"RawMessageDelivery": "true"}'
aws sns subscribe --topic-arn $TOPIC --protocol sqs \
  --notification-endpoint arn:aws:sqs:ap-south-1:111122223333:audit-queue

{
  "source_system": ["core-banking", "cards"],
  "file_type": [{ "prefix": "txn" }],
  "size_mb": [{ "numeric": [">", 0] }]
}

A subscription filter policy (above, applied to message attributes by default, or to the message body) delivers only matching messages to that subscriber, so each queue receives only what it needs.

import json, boto3

boto3.client("sns").publish(
    TopicArn="arn:aws:sns:ap-south-1:111122223333:file-arrivals",
    Message=json.dumps({"bucket": "acme-lake-prod", "key": "raw/txn/2026-10-26.csv.gz"}),
    MessageAttributes={
        "source_system": {"DataType": "String", "StringValue": "core-banking"},
        "file_type":     {"DataType": "String", "StringValue": "txn_daily"},
        "size_mb":       {"DataType": "Number", "StringValue": "512"},
    },
)

3. Pipeline Alerting

 CloudWatch alarm (Glue job failed, DLQ > 0, Lambda errors)  ─┐
 Step Functions "NotifyFailure" state                         ├──► SNS topic "pipeline-alerts" ──► email / on-call (HTTPS)
 EventBridge rule (Glue Job State Change = FAILED)            ─┘                                └─► chat (Slack/Teams via AWS chat integration)

• Put the DAG/job name, run date, error and a log link in every message.
• Use separate topics for alerts and informational events, so on-call is not paged for successes.
• Encrypt topics with KMS and restrict who can publish with topic policies.
• Failed deliveries to subscribers can go to a subscription DLQ (an SQS queue).

4. Amazon EventBridge

EventBridge is a serverless event bus: AWS services, SaaS apps and your code publish events, and rules match event patterns and route them to targets (Step Functions, Lambda, SQS, SNS, Kinesis, API destinations and more).

{
  "source": ["aws.glue"],
  "detail-type": ["Glue Job State Change"],
  "detail": {
    "jobName": [{ "prefix": "raw-to-silver" }],
    "state": ["FAILED", "TIMEOUT"]
  }
}

# publish your own business events
import json, boto3
boto3.client("events").put_events(Entries=[{
    "EventBusName": "data-platform",
    "Source": "acme.pipelines",
    "DetailType": "Partition Ready",
    "Detail": json.dumps({"table": "silver.transactions", "dt": "2026-10-26", "rows": 4810233}),
}])

5. SNS vs SQS vs EventBridge vs Kinesis

6. Common Interview Questions