AWS Lambda for Data Engineers

Limits and invocation models, S3/SQS/Kinesis handler examples, packaging and deployment, configuration and secrets, idempotency, observability and anti-patterns.

1. What AWS Lambda Is

AWS Lambda runs your function in response to events without servers to manage. You pay per request and per GB-second of execution. In data engineering it is the glue for short, event-driven steps: validate a file, start a job, route a message, call an API, send an alert.

2. Invocation Models

3. Example: Validate a File and Start a Pipeline

import json, os, urllib.parse
import boto3

s3 = boto3.client("s3")
sfn = boto3.client("stepfunctions")
STATE_MACHINE_ARN = os.environ["STATE_MACHINE_ARN"]  # configuration via environment variables
MAX_BYTES = 5 * 1024 ** 3

def handler(event, context):
    for record in event["Records"]:  # one S3 event can contain several records
        bucket = record["s3"]["bucket"]["name"]
        key = urllib.parse.unquote_plus(record["s3"]["object"]["key"])  # keys arrive URL-encoded
        head = s3.head_object(Bucket=bucket, Key=key)

        if not key.endswith(".csv.gz") or head["ContentLength"] == 0 or head["ContentLength"] > MAX_BYTES:
            s3.copy_object(Bucket=bucket, Key=f"_quarantine/{key}", CopySource={"Bucket": bucket, "Key": key})
            print(json.dumps({"level": "WARN", "msg": "rejected file", "key": key}))
            continue

        sfn.start_execution(
            stateMachineArn=STATE_MACHINE_ARN,
            name=f"load-{head['ETag'].strip(chr(34))}",  # same file → same name → no duplicate run
            input=json.dumps({"bucket": bucket, "key": key, "size": head["ContentLength"]}),
        )
    return {"status": "ok"}

Using the object ETag in the execution name makes the trigger idempotent: if S3 delivers the same event twice, Step Functions rejects the duplicate name for a Standard workflow.

4. Example: Batch Processing from SQS

import json

def handler(event, context):
    failures = []
    for record in event["Records"]:
        try:
            msg = json.loads(record["body"])
            process(msg)  # your logic; must be idempotent
        except Exception as exc:
            print(f"failed {record['messageId']}: {exc}")
            failures.append({"itemIdentifier": record["messageId"]})
  # only the failed messages return to the queue; the rest are deleted
    return {"batchItemFailures": failures}

Set the SQS visibility timeout to at least six times the function timeout, or messages reappear while still being processed.

5. Example: Kinesis Stream Consumer

import base64, json

def handler(event, context):
    rows = []
    for record in event["Records"]:
        payload = base64.b64decode(record["kinesis"]["data"])  # data arrives base64-encoded
        txn = json.loads(payload)
        if txn.get("amount", 0) > 100000:
            rows.append(txn)
    if rows:
        write_alerts(rows)  # e.g. DynamoDB batch_writer
    return {"batchItemFailures": []}

6. Packaging & Deployment

# zip deployment with dependencies
pip install -r requirements.txt -t package/
cp handler.py package/ && (cd package && zip -r ../function.zip .)
aws lambda create-function --function-name file-validator \
  --runtime python3.12 --handler handler.handler --architectures arm64 \
  --role arn:aws:iam::111122223333:role/file-validator-role \
  --memory-size 512 --timeout 60 --zip-file fileb://function.zip \
  --environment "Variables={STATE_MACHINE_ARN=arn:aws:states:ap-south-1:111122223333:stateMachine:daily-load}"

aws lambda update-function-code --function-name file-validator --zip-file fileb://function.zip
aws lambda invoke --function-name file-validator --payload fileb://test_event.json out.json

• Layers share dependencies across functions; AWS publishes a managed layer for the AWS SDK for pandas (pandas, pyarrow, awswrangler).
• Container images (up to 10 GB) for heavy dependencies or native libraries.
• Deploy with infrastructure as code (SAM, CDK, CloudFormation, Terraform) rather than the console.
• arm64 (Graviton) functions are typically about 20% cheaper with similar or better performance.

7. Configuration, Secrets & Networking

8. Idempotency, Observability & Anti-Patterns

• Every event source can deliver the same event more than once, so make handlers idempotent: conditional writes, deterministic IDs, or the Powertools for AWS Lambda idempotency utility backed by DynamoDB.
• Use Powertools for AWS Lambda (Python) for structured logging, metrics and tracing; enable X-Ray for request traces.
• Alarm on Errors, Throttles, Duration near the timeout, IteratorAge (streams) and DLQ depth.

9. Common Interview Questions