Amazon EMR for Data Engineers

EMR on EC2, Serverless and EKS; node types and instance fleets; creating clusters and steps; Glue Catalog integration; EMR Serverless jobs; tuning, security and monitoring.

1. What Amazon EMR Is

Amazon EMR runs open-source big data frameworks (Spark, Hive, Trino/Presto, Flink, HBase, Hudi, Iceberg, Delta) on AWS-managed infrastructure. Compared with Glue, it gives full control over versions, configuration, instance types and tuning, and is usually cheaper for large, steady workloads (especially with spot instances).

2. Cluster Architecture

• Keep data in S3 (EMRFS) rather than HDFS so clusters can be transient: created for a pipeline run and terminated afterwards.
• Instance fleets list several instance types per node type with a spot/on-demand mix, so EMR can find capacity and reduce spot interruptions.
• Managed scaling resizes the cluster from YARN metrics; auto-termination shuts down idle clusters.

3. Creating a Cluster & Submitting Steps

aws emr create-cluster \
  --name "nightly-enrichment" \
  --release-label emr-7.5.0 \
  --applications Name=Spark Name=Hive \
  --service-role EMR_DefaultRole_V2 \
  --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole,SubnetIds=subnet-0abc123 \
  --instance-fleets file://fleets.json \
  --managed-scaling-policy '{"ComputeLimits": {"UnitType": "InstanceFleetUnits", "MinimumCapacityUnits": 4, "MaximumCapacityUnits": 40}}' \
  --auto-termination-policy IdleTimeout=1800 \
  --log-uri s3://acme-emr-logs/ \
  --bootstrap-actions Path=s3://acme-code/emr/bootstrap.sh,Name=install-deps \
  --configurations file://configurations.json \
  --steps Type=Spark,Name=enrich,ActionOnFailure=TERMINATE_CLUSTER,Args=[--deploy-mode,cluster,s3://acme-code/jobs/enrich.py,--run-date,2026-10-26] \
  --auto-terminate  # terminate after the steps finish (transient cluster)

[
  {
    "Classification": "spark-hive-site",
    "Properties": {
      "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
    }
  },
  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.sql.adaptive.enabled": "true",
      "spark.dynamicAllocation.enabled": "true",
      "spark.sql.shuffle.partitions": "400"
    }
  }
]

The spark-hive-site classification makes Spark use the Glue Data Catalog as its metastore, so tables are shared with Athena and Glue.

# add work to a running cluster
aws emr add-steps --cluster-id j-2AXXXXXXGAPLF --steps \
  Type=CUSTOM_JAR,Name=load,Jar=command-runner.jar,ActionOnFailure=CONTINUE,Args=[spark-submit,--deploy-mode,cluster,s3://acme-code/jobs/load.py]

aws emr list-steps --cluster-id j-2AXXXXXXGAPLF --step-states FAILED
aws emr terminate-clusters --cluster-ids j-2AXXXXXXGAPLF

4. EMR Serverless

aws emr-serverless create-application --name spark-etl --type SPARK --release-label emr-7.5.0 \
  --maximum-capacity '{"cpu": "200 vCPU", "memory": "800 GB"}' \
  --auto-stop-configuration '{"enabled": true, "idleTimeoutMinutes": 15}'

aws emr-serverless start-job-run \
  --application-id 00fabcdexample \
  --execution-role-arn arn:aws:iam::111122223333:role/emr-serverless-job \
  --name enrich-2026-10-26 \
  --job-driver '{"sparkSubmit": {
      "entryPoint": "s3://acme-code/jobs/enrich.py",
      "entryPointArguments": ["--run-date", "2026-10-26"],
      "sparkSubmitParameters": "--conf spark.executor.cores=4 --conf spark.executor.memory=16g --conf spark.dynamicAllocation.maxExecutors=50"}}' \
  --configuration-overrides '{"monitoringConfiguration": {"s3MonitoringConfiguration": {"logUri": "s3://acme-emr-logs/serverless/"}}}'

• You pay for vCPU, memory and storage used by the job run, with no idle cluster cost.
• Pre-initialised capacity keeps warm workers for jobs that must start in seconds.
• Orchestrate with Step Functions (emr-serverless:startJobRun.sync) or Airflow (EmrServerlessStartJobOperator).

5. Spark Tuning & Table Formats on EMR

• EMR's Spark runtime is optimised and API-compatible with open-source Spark; newer release labels bring speed-ups.
• maximizeResourceAllocation (spark classification) sizes executors to fill nodes: a quick start, but explicit sizing (about 5 cores per executor) is better for production. See the Spark guide.
• Use Graviton (m7g/r7g) instances for better price-performance, and r-family for memory-heavy joins.
• Iceberg, Hudi and Delta are included: enable with a classification such as {"Classification": "iceberg-defaults", "Properties": {"iceberg.enabled": "true"}}.
• Write intermediate shuffle data to local instance storage (d-family instances) for heavy shuffles.

6. Security & Monitoring

7. Common Interview Questions