AWS CloudFormation for Data Engineers

Templates, stacks and change sets, a full data-lake stack, intrinsic functions and dynamic references, safe deployments, CDK and SAM, and how they compare with Terraform.

1. CloudFormation Concepts

AWS CloudFormation provisions AWS resources from templates (YAML or JSON). A deployed template is a stack; CloudFormation works out the order of creation, rolls back on failure, and tracks what it manages, so environments are reproducible and changes are reviewed as code.

2. Example: A Data Lake Stack

AWSTemplateFormatVersion: "2010-09-09"
Description: Data lake bucket, Glue database and ETL job

Parameters:
  Env:
    Type: String
    AllowedValues: [dev, test, prod]
  ScriptBucket:
    Type: String

Conditions:
  IsProd: !Equals [!Ref Env, prod]

Resources:
  LakeKey:
    Type: AWS::KMS::Key
    Properties:
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: AllowAccountAdmin
            Effect: Allow
            Principal: { AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root" }
            Action: "kms:*"
            Resource: "*"

  LakeBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain  # never delete data with the stack
    UpdateReplacePolicy: Retain
    Properties:
      BucketName: !Sub "acme-lake-${Env}-${AWS::AccountId}"
      VersioningConfiguration: { Status: Enabled }
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: aws:kms
              KMSMasterKeyID: !GetAtt LakeKey.Arn
            BucketKeyEnabled: true
      LifecycleConfiguration:
        Rules:
          - Id: raw-to-ia
            Status: Enabled
            Prefix: raw/
            Transitions:
              - { StorageClass: STANDARD_IA, TransitionInDays: 30 }

  SilverDatabase:
    Type: AWS::Glue::Database
    Properties:
      CatalogId: !Ref AWS::AccountId
      DatabaseInput: { Name: !Sub "silver_${Env}" }

  GlueRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: { Service: glue.amazonaws.com }
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
      Policies:
        - PolicyName: lake-access
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: [s3:GetObject, s3:PutObject, s3:DeleteObject]
                Resource: !Sub "${LakeBucket.Arn}/*"
              - Effect: Allow
                Action: s3:ListBucket
                Resource: !GetAtt LakeBucket.Arn

  RawToSilverJob:
    Type: AWS::Glue::Job
    Properties:
      Name: !Sub "raw-to-silver-${Env}"
      Role: !GetAtt GlueRole.Arn
      GlueVersion: "5.0"
      WorkerType: G.1X
      NumberOfWorkers: !If [IsProd, 20, 4]
      Command:
        Name: glueetl
        PythonVersion: "3"
        ScriptLocation: !Sub "s3://${ScriptBucket}/glue/raw_to_silver.py"
      DefaultArguments:
        "--job-bookmark-option": job-bookmark-enable
        "--enable-metrics": "true"

Outputs:
  LakeBucketName:
    Value: !Ref LakeBucket
    Export: { Name: !Sub "${AWS::StackName}-LakeBucket" }

3. Intrinsic Functions & Dynamic References

# dynamic references resolve secrets and parameters at deploy time (never hard-code them)
MasterUserPassword: "{{resolve:secretsmanager:prod/orders-db:SecretString:password}}"
ImageTag: "{{resolve:ssm:/data-platform/etl-image-tag}}"

4. Deploying & Reviewing Changes

cfn-lint data-lake.yaml  # lint before anything else
aws cloudformation validate-template --template-body file://data-lake.yaml

aws cloudformation deploy \
  --template-file data-lake.yaml \
  --stack-name data-lake-prod \
  --parameter-overrides Env=prod ScriptBucket=acme-code-prod \
  --capabilities CAPABILITY_NAMED_IAM \
  --tags owner=data-platform costcentre=fraud \
  --no-execute-changeset  # create a change set to review first

aws cloudformation describe-change-set --stack-name data-lake-prod --change-set-name <name>
aws cloudformation execute-change-set --stack-name data-lake-prod --change-set-name <name>

aws cloudformation describe-stack-events --stack-name data-lake-prod --max-items 20  # why did it fail?
aws cloudformation detect-stack-drift --stack-name data-lake-prod

5. AWS CDK & SAM

The AWS CDK lets you define infrastructure in Python (or TypeScript, Java…) with loops, functions and sensible defaults; it synthesises CloudFormation templates.

# app.py: pip install aws-cdk-lib constructs ; deploy with: cdk deploy
from aws_cdk import App, Stack, RemovalPolicy, Duration
from aws_cdk import aws_s3 as s3, aws_lambda as _lambda, aws_s3_notifications as s3n
from constructs import Construct

class IngestStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        bucket = s3.Bucket(self, "Inbound",
            encryption=s3.BucketEncryption.KMS_MANAGED,
            block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
            enforce_ssl=True, versioned=True,
            removal_policy=RemovalPolicy.RETAIN)

        validator = _lambda.Function(self, "Validator",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="handler.handler",
            code=_lambda.Code.from_asset("lambda/validator"),
            timeout=Duration.seconds(60))

        bucket.grant_read(validator)  # least-privilege policy generated for you
        bucket.add_event_notification(s3.EventType.OBJECT_CREATED, s3n.LambdaDestination(validator),
                                      s3.NotificationKeyFilter(prefix="incoming/", suffix=".csv"))

app = App()
IngestStack(app, "ingest-dev")
app.synth()

AWS SAM is a CloudFormation extension with short syntax for serverless resources (functions, APIs, event sources) and a CLI to build, test locally and deploy (sam build, sam local invoke, sam deploy --guided).

6. Common Interview Questions