Amazon DynamoDB for Data Engineers

Keys and capacity, access-pattern modelling, boto3 examples (conditional writes, queries, watermarks), streams and exports, transactions, TTL, backups and cost.

1. DynamoDB Concepts

Amazon DynamoDB is a serverless key-value and document database with single-digit millisecond latency at any scale. Data engineers use it as a source (application data, exported to the lake) and as pipeline infrastructure: watermarks, idempotency keys, job state, configuration and online feature serving.

2. Data Modelling: Access Patterns First

DynamoDB has no joins and no ad-hoc queries: design keys from the questions the application asks. A common approach is single-table design, storing related entities together so one query returns them:

• Hot partitions: one very busy key value is throttled even if the table has spare capacity. Spread writes (e.g. add a suffix shard to the key) for hot keys.
• Sparse GSIs: only items with the GSI key attribute appear in the index, e.g. index only items with status = OPEN.
• Use DynamoDB for known access patterns; send analytics to S3/Athena/Redshift via exports or streams.

3. Working with DynamoDB in Python

import boto3
from boto3.dynamodb.conditions import Key
from botocore.exceptions import ClientError

table = boto3.resource("dynamodb").Table("fraud-app")

# idempotent insert: only if this key does not exist yet
try:
    table.put_item(
        Item={"PK": "ACCOUNT#A1", "SK": "TXN#2026-10-26T10:15:00#T9", "amount": 2500, "channel": "UPI"},
        ConditionExpression="attribute_not_exists(PK)",
    )
except ClientError as e:
    if e.response["Error"]["Code"] != "ConditionalCheckFailedException":
        raise  # anything else is a real error; duplicate is fine

# query: last 20 transactions for an account, newest first
resp = table.query(
    KeyConditionExpression=Key("PK").eq("ACCOUNT#A1") & Key("SK").begins_with("TXN#"),
    ScanIndexForward=False, Limit=20)
items = resp["Items"]

# atomic counter / update
table.update_item(
    Key={"PK": "ACCOUNT#A1", "SK": "PROFILE"},
    UpdateExpression="SET txn_count = if_not_exists(txn_count, :zero) + :one, last_seen = :ts",
    ExpressionAttributeValues={":zero": 0, ":one": 1, ":ts": "2026-10-26T10:15:00Z"})

# bulk writes (handles batching and retries of unprocessed items)
with table.batch_writer() as batch:
    for row in rows:
        batch.put_item(Item=row)

# pipeline watermark table: read, process, then advance only if nobody else did
state = boto3.resource("dynamodb").Table("pipeline-state")
current = state.get_item(Key={"pipeline": "daily_txn"}, ConsistentRead=True).get("Item", {})
old_wm = current.get("watermark", "1970-01-01T00:00:00Z")
new_wm = run_incremental_load(since=old_wm)
state.update_item(
    Key={"pipeline": "daily_txn"},
    UpdateExpression="SET watermark = :new",
    ConditionExpression="attribute_not_exists(watermark) OR watermark = :old",  # optimistic lock
    ExpressionAttributeValues={":new": new_wm, ":old": old_wm})

Avoid Scan in application paths: it reads the whole table. Paginate with LastEvaluatedKey when you must scan (e.g. small admin tasks).

4. Streams, Exports & Analytics

aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:ap-south-1:111122223333:table/fraud-app \
  --s3-bucket acme-lake-prod --s3-prefix raw/dynamodb/fraud-app/ \
  --export-format DYNAMODB_JSON \
  --export-type INCREMENTAL_EXPORT \
  --incremental-export-specification "ExportFromTime=2026-10-25T00:00:00Z,ExportToTime=2026-10-26T00:00:00Z,ExportViewType=NEW_AND_OLD_IMAGES"

5. Transactions, TTL, Backups & Cost

• Transactions (TransactWriteItems): all-or-nothing writes across up to 100 items, e.g. debit one account and credit another.
• TTL: set an epoch-seconds attribute and DynamoDB deletes expired items for free (useful for idempotency keys and sessions).
• Point-in-time recovery: restore to any second in the last 35 days; on-demand backups for long-term copies.
• Global tables: multi-region, multi-active replication.
• DAX: in-memory cache for microsecond reads on read-heavy tables.
• Cost: on-demand is simplest; switch steady tables to provisioned with auto scaling. Keep items small, project only needed attributes into GSIs, and use TTL to drop stale data.

6. Common Interview Questions