AWS Glue for Data Engineers

The Data Catalog, crawlers, a full PySpark ETL job, job settings, bookmarks, Iceberg writes, parallel JDBC reads, Data Quality, tuning and monitoring.

1. What AWS Glue Is

AWS Glue is a serverless data integration service. It combines a central metadata catalog with managed Spark (and Python) jobs, so you can discover, catalog, transform and load data without managing clusters.

2. The Data Catalog

aws glue create-database --database-input '{"Name": "silver", "Description": "Cleaned and conformed data"}'
aws glue get-tables --database-name silver --query "TableList[].Name"
aws glue get-partitions --database-name silver --table-name transactions --max-items 5

import boto3
glue = boto3.client("glue")

# register a new partition yourself (cheaper and more predictable than re-running a crawler)
table = glue.get_table(DatabaseName="silver", Name="transactions")["Table"]
sd = dict(table["StorageDescriptor"])
sd["Location"] = "s3://acme-lake-prod/silver/transactions/dt=2026-10-26/"
glue.create_partition(DatabaseName="silver", TableName="transactions",
                      PartitionInput={"Values": ["2026-10-26"], "StorageDescriptor": sd})

• Tables can also be created with Athena DDL (CREATE EXTERNAL TABLE), CloudFormation/Terraform, or by Spark/Glue jobs writing with enableUpdateCatalog.
• For tables with many partitions, add partition indexes or use Athena partition projection so queries do not list every partition.
• EMR and Databricks can use the Glue Catalog as their Hive metastore, giving one catalog across engines.

3. Crawlers

aws glue create-crawler --name raw-transactions-crawler \
  --role arn:aws:iam::111122223333:role/glue-crawler-role \
  --database-name raw \
  --targets '{"S3Targets": [{"Path": "s3://acme-lake-prod/raw/transactions/"}]}' \
  --schedule "cron(15 2 * * ? *)" \
  --recrawl-policy '{"RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY"}' \
  --schema-change-policy '{"UpdateBehavior": "LOG", "DeleteBehavior": "LOG"}'

aws glue start-crawler --name raw-transactions-crawler

Crawlers are great for discovery and messy raw zones. For stable curated tables, define the schema in code and add partitions explicitly: fewer surprises and lower cost.

4. A Glue ETL Job (PySpark)

import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame
from awsglue.transforms import ApplyMapping
from awsglue.job import Job
from pyspark.context import SparkContext
from pyspark.sql import functions as F

args = getResolvedOptions(sys.argv, ["JOB_NAME", "run_date"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# read only one partition from the catalog (push-down predicate prunes the S3 listing)
src = glueContext.create_dynamic_frame.from_catalog(
    database="raw", table_name="transactions",
    push_down_predicate=f"ingest_date = '{args['run_date']}'",
    transformation_ctx="src")  # transformation_ctx enables job bookmarks

mapped = ApplyMapping.apply(frame=src, mappings=[
    ("txn_id", "string", "txn_id", "string"),
    ("account_id", "string", "account_id", "string"),
    ("amount", "string", "amount", "decimal(18,2)"),
    ("event_ts", "string", "event_ts", "timestamp"),
], transformation_ctx="mapped")

df = (mapped.toDF()  # switch to a Spark DataFrame for logic
      .dropDuplicates(["txn_id"])
      .filter(F.col("amount").isNotNull())
      .withColumn("dt", F.to_date("event_ts")))

glueContext.write_dynamic_frame.from_options(
    frame=DynamicFrame.fromDF(df, glueContext, "out"),
    connection_type="s3",
    connection_options={"path": "s3://acme-lake-prod/silver/transactions/", "partitionKeys": ["dt"]},
    format="glueparquet",
    transformation_ctx="sink")

job.commit()  # saves bookmark state

aws glue create-job --name raw-to-silver-transactions \
  --role arn:aws:iam::111122223333:role/glue-etl-role \
  --glue-version "5.0" --worker-type G.1X --number-of-workers 10 \
  --command '{"Name": "glueetl", "ScriptLocation": "s3://acme-code/glue/raw_to_silver.py", "PythonVersion": "3"}' \
  --default-arguments '{"--job-bookmark-option": "job-bookmark-enable", "--enable-metrics": "true",
                        "--enable-continuous-cloudwatch-log": "true", "--enable-auto-scaling": "true"}'

aws glue start-job-run --job-name raw-to-silver-transactions --arguments '{"--run_date": "2026-10-26"}'
aws glue get-job-runs --job-name raw-to-silver-transactions --max-items 3

5. Job Settings, Workers & Special Parameters

6. Incremental Processing & Job Bookmarks

Job bookmarks remember what a job already processed, so the next run reads only new data: new S3 files (by modification time) or new JDBC rows (by a monotonically increasing key column).

• Every source and sink needs a transformation_ctx, and the script must call job.commit(); otherwise state is not saved.
• Bookmarks are per job: renaming the job or resetting the bookmark (aws glue reset-job-bookmark --job-name …) reprocesses everything.
• Bookmarks do not handle updates to files already seen. For updates and deletes, use CDC files with MERGE into Iceberg/Delta/Hudi, or your own watermark in a control table (DynamoDB).
• Use push_down_predicate to restrict reads to the partitions a run should touch, and catalogPartitionPredicate with partition indexes for very large tables.

7. Writing Iceberg Tables from Glue

# job parameters:
#   --datalake-formats  iceberg
#   --conf  spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
#           --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog
#           --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
#           --conf spark.sql.catalog.glue_catalog.warehouse=s3://acme-lake-prod/warehouse/
#           --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO

df.createOrReplaceTempView("updates")
spark.sql("""
    MERGE INTO glue_catalog.silver.customers t
    USING updates s
    ON t.customer_id = s.customer_id
    WHEN MATCHED AND s.op = 'D' THEN DELETE
    WHEN MATCHED THEN UPDATE SET *
    WHEN NOT MATCHED AND s.op <> 'D' THEN INSERT *
""")

8. Reading Databases (JDBC) in Parallel

# connection "orders-postgres" holds the JDBC URL, VPC subnet and a Secrets Manager secret
orders = glueContext.create_dynamic_frame.from_options(
    connection_type="postgresql",
    connection_options={
        "useConnectionProperties": "true",
        "connectionName": "orders-postgres",
        "dbtable": "public.orders",
        "hashfield": "order_id",  # split the read on this column…
        "hashpartitions": "16",  # …into 16 parallel queries
    },
    transformation_ctx="orders")

• Without hashfield/hashpartitions a JDBC read runs as a single query on one executor, which is the classic reason a Glue database extract is slow.
• Glue needs network access to the database: the connection places workers in your VPC subnet with a security group the database allows.
• For continuous change capture rather than repeated full reads, use AWS DMS and process its CDC files.

9. Glue Data Quality

from awsgluedq.transforms import EvaluateDataQuality

ruleset = """
Rules = [
    RowCount > 0,
    IsComplete "txn_id",
    IsUnique "txn_id",
    ColumnValues "amount" > 0,
    ColumnValues "channel" in ["UPI", "CARD_POS", "ATM", "NEFT"],
    Completeness "account_id" > 0.99
]
"""

dq_results = EvaluateDataQuality().process_rows(
    frame=DynamicFrame.fromDF(df, glueContext, "dq_input"),
    ruleset=ruleset,
    publishing_options={"dataQualityEvaluationContext": "silver_txn", "enableDataQualityResultsPublishing": True},
)

Results are published to CloudWatch and EventBridge, so a failed ruleset can stop a Step Functions pipeline or send an alert. Rules can also run directly against catalog tables on a schedule, without a job.

10. Performance, Cost & Monitoring

• Enable the Spark UI (event logs to S3) and job run insights for root-cause hints.
• Continuous logging to CloudWatch; metrics such as glue.driver.aggregate.bytesRead and executor memory.
• Develop interactively with Glue interactive sessions or Docker images of the Glue libraries, then deploy scripts via CI/CD.

11. Common Interview Questions