Azure Reference

Azure for data engineers: identity and RBAC, ADLS Gen2, Data Factory, Azure Databricks, Synapse, Fabric, Event Hubs and Stream Analytics, Cosmos DB, Key Vault and Purview, monitoring, Bicep and CI/CD, with a reference architecture and AWS/GCP mapping.

1. Azure Fundamentals

Everything in Azure lives in a hierarchy. Policies and access granted at a level are inherited by everything below it:

 Microsoft Entra ID tenant (identities: users, groups, apps)
   └── Management groups      (e.g. "Bank" → "Prod", "NonProd"; Azure Policy applied here)
         └── Subscriptions    (billing + quota boundary: sub-data-prod, sub-data-dev)
               └── Resource groups   (lifecycle unit: rg-lakehouse-prod)
                     └── Resources   (storage accounts, Data Factory, Databricks workspace …)

az login                                         # browser / device-code sign-in
az account list -o table
az account set --subscription "sub-data-dev"
az group create --name rg-lakehouse-dev --location centralindia \
  --tags env=dev owner=data-platform costcentre=fraud
az resource list -g rg-lakehouse-dev -o table

2. Identity & Access (Entra ID, RBAC, Managed Identities)

Azure RBAC = who (security principal) + what (role definition) + where (scope: management group, subscription, resource group or resource).

# let Data Factory's managed identity write to the lake (data-plane role, scoped to one account)
ADF_MI=$(az datafactory show -g rg-lakehouse-dev -n adf-acme-dev --query identity.principalId -o tsv)
LAKE_ID=$(az storage account show -g rg-lakehouse-dev -n stacmelakedev --query id -o tsv)

az role assignment create --assignee-object-id "$ADF_MI" --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" --scope "$LAKE_ID"

Classic trap: a user with Contributor on a storage account still gets "authorization permission mismatch" reading blobs with Entra ID auth, because Contributor is a control-plane role. Data access needs a Storage Blob Data role (or ACLs on ADLS Gen2).

3. Storage: ADLS Gen2 & Blob Storage

Azure Data Lake Storage Gen2 is a Blob Storage account with the hierarchical namespace enabled: real directories, atomic directory renames, and POSIX-style ACLs, which is what Spark and Hadoop need. It is the standard lake storage on Azure.

az storage account create -n stacmelakedev -g rg-lakehouse-dev -l centralindia \
  --sku Standard_ZRS --kind StorageV2 --hns true \
  --min-tls-version TLS1_2 --allow-blob-public-access false

for c in raw bronze silver gold; do
  az storage fs create -n $c --account-name stacmelakedev --auth-mode login
done

az storage fs directory create -f raw -n landing/transactions --account-name stacmelakedev --auth-mode login
az storage fs file upload -f raw -s ./txn_2026-10-26.json \
  -p landing/transactions/txn_2026-10-26.json --account-name stacmelakedev --auth-mode login

Paths use the abfss:// scheme: abfss://<container>@<account>.dfs.core.windows.net/<path>.

{
  "rules": [{
    "enabled": true, "name": "raw-tiering", "type": "Lifecycle",
    "definition": {
      "filters": { "blobTypes": ["blockBlob"], "prefixMatch": ["raw/landing/"] },
      "actions": { "baseBlob": {
        "tierToCool":    { "daysAfterModificationGreaterThan": 30 },
        "tierToArchive": { "daysAfterModificationGreaterThan": 180 },
        "delete":        { "daysAfterModificationGreaterThan": 2555 } } }
    }
  }]
}

Apply with az storage account management-policy create --account-name stacmelakedev -g rg-lakehouse-dev --policy @policy.json. Combine with soft delete, versioning and immutability (WORM) policies for records that must be retained.

Reading ADLS from Spark

# Open-source Spark / Synapse with a service principal (OAuth). On Databricks prefer Unity Catalog.
acct = "stacmelakedev.dfs.core.windows.net"
spark.conf.set(f"fs.azure.account.auth.type.{acct}", "OAuth")
spark.conf.set(f"fs.azure.account.oauth.provider.type.{acct}",
               "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set(f"fs.azure.account.oauth2.client.id.{acct}", client_id)
spark.conf.set(f"fs.azure.account.oauth2.client.secret.{acct}", client_secret)   # from Key Vault
spark.conf.set(f"fs.azure.account.oauth2.client.endpoint.{acct}",
               f"https://login.microsoftonline.com/{tenant_id}/oauth2/token")

df = spark.read.json("abfss://raw@stacmelakedev.dfs.core.windows.net/landing/transactions/")
df.write.format("delta").mode("append").save("abfss://bronze@stacmelakedev.dfs.core.windows.net/transactions")

4. Azure Data Factory

Azure Data Factory (ADF) is Azure's managed orchestration and data-movement service: 90+ connectors, visual pipelines, and scheduling. It moves and orchestrates; heavy transformation is usually delegated to Databricks, Synapse or SQL.

Expressions

@pipeline().parameters.runDate                         -- a pipeline parameter
@pipeline().RunId                                      -- unique id of this run
@trigger().outputs.windowStartTime                     -- tumbling-window start
@activity('LookupWatermark').output.firstRow.last_wm   -- output of an earlier activity
@formatDateTime(utcNow(), 'yyyy/MM/dd')                -- date folders
@concat('landing/orders/', formatDateTime(pipeline().parameters.runDate, 'yyyy-MM-dd'))
"SELECT * FROM dbo.orders WHERE updated_at > '@{activity('LookupWatermark').output.firstRow.last_wm}'"
                                                       -- @{ } interpolates inside a string

Incremental copy with a watermark

 LookupOldWatermark ──► LookupNewWatermark ──► CopyChangedRows ──► UpdateWatermark
 (SELECT last_wm FROM    (SELECT MAX(updated_at)   (source query between       (stored procedure writes
  etl.watermarks)         AS new_wm FROM orders)    old and new watermark)      new_wm back)

{
  "name": "CopyChangedRows",
  "type": "Copy",
  "dependsOn": [
    { "activity": "LookupOldWatermark", "dependencyConditions": ["Succeeded"] },
    { "activity": "LookupNewWatermark", "dependencyConditions": ["Succeeded"] }
  ],
  "typeProperties": {
    "source": {
      "type": "AzureSqlSource",
      "sqlReaderQuery": {
        "value": "SELECT * FROM dbo.orders WHERE updated_at > '@{activity('LookupOldWatermark').output.firstRow.last_wm}' AND updated_at <= '@{activity('LookupNewWatermark').output.firstRow.new_wm}'",
        "type": "Expression"
      }
    },
    "sink": { "type": "ParquetSink" }
  },
  "inputs":  [ { "referenceName": "ds_sql_orders", "type": "DatasetReference" } ],
  "outputs": [ { "referenceName": "ds_adls_orders_parquet", "type": "DatasetReference",
                 "parameters": { "folder": "@formatDateTime(pipeline().parameters.runDate, 'yyyy/MM/dd')" } } ]
}

• Metadata-driven pipelines: keep a control table of sources (table, watermark column, target path) and loop over it with Lookup + ForEach, so one pipeline loads hundreds of tables.
• CI/CD: connect the dev factory to Git; collaboration happens in feature branches; publishing generates ARM templates, which a release pipeline deploys to test and prod with environment-specific parameters. Only dev is edited by hand.
• Monitoring: the Monitor hub shows runs; send diagnostic logs to Log Analytics and alert on failed runs.

5. Azure Databricks

Azure Databricks is Databricks as a first-party Azure service, billed through Azure and integrated with Entra ID, ADLS, Key Vault and VNets. Everything in the Spark & Databricks reference applies; this is the Azure-specific wiring.

 Unity Catalog metastore (one per region)
   └── Storage credential  ──uses──►  Access Connector for Azure Databricks (managed identity)
                                         └─ granted "Storage Blob Data Contributor" on the ADLS account
   └── External location   abfss://raw@stacmelakedev.dfs.core.windows.net/landing
   └── Catalogs → schemas → tables / volumes (managed storage on ADLS)

-- after creating the storage credential from the access connector (UI or API)
CREATE EXTERNAL LOCATION IF NOT EXISTS raw_landing
  URL 'abfss://raw@stacmelakedev.dfs.core.windows.net/landing'
  WITH (STORAGE CREDENTIAL acme_lake_mi);

GRANT READ FILES ON EXTERNAL LOCATION raw_landing TO `data-engineers`;

CREATE CATALOG IF NOT EXISTS fraud_dev
  MANAGED LOCATION 'abfss://managed@stacmelakedev.dfs.core.windows.net/fraud_dev';

# secrets from an Azure Key Vault-backed secret scope (values are redacted in notebook output)
sql_password = dbutils.secrets.get(scope="kv-acme-dev", key="sql-etl-password")

# Auto Loader straight from ADLS (governed by the external location)
(spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", "/Volumes/fraud_dev/bronze/chk/txn_schema")
    .load("abfss://raw@stacmelakedev.dfs.core.windows.net/landing/transactions/")
    .writeStream.option("checkpointLocation", "/Volumes/fraud_dev/bronze/chk/txn")
    .trigger(availableNow=True)
    .toTable("fraud_dev.bronze.transactions"))

Mount points (dbutils.fs.mount) and account keys in notebooks are legacy patterns. Use Unity Catalog external locations and managed identities instead. For private networking, deploy the workspace into your own VNet (VNet injection) with Private Link and secure cluster connectivity (no public IPs).

6. Azure Synapse Analytics

Synapse combines an MPP data warehouse, serverless SQL over the lake, Spark pools and ADF-style pipelines in one workspace. Microsoft now positions Fabric as the successor for new analytics projects, but Synapse remains widely used in existing estates and interviews.

-- Dedicated pool: choose a distribution per table
CREATE TABLE dbo.fact_sales (
  sale_id BIGINT, customer_id INT, date_id INT, amount DECIMAL(18,2))
WITH (DISTRIBUTION = HASH(customer_id), CLUSTERED COLUMNSTORE INDEX);

CREATE TABLE dbo.dim_date (date_id INT, full_date DATE, month_name VARCHAR(12))
WITH (DISTRIBUTION = REPLICATE);          -- small dimension copied to every compute node

-- fastest load path: COPY INTO from the lake using the workspace managed identity
COPY INTO dbo.fact_sales
FROM 'https://stacmelakedev.blob.core.windows.net/gold/fact_sales/*.parquet'
WITH (FILE_TYPE = 'PARQUET', CREDENTIAL = (IDENTITY = 'Managed Identity'));

-- Serverless pool: query the lake in place, pay per TB scanned
SELECT channel, COUNT(*) AS txns, SUM(amount) AS total
FROM OPENROWSET(
       BULK 'https://stacmelakedev.dfs.core.windows.net/silver/transactions/',
       FORMAT = 'DELTA') AS t
WHERE event_date = '2026-10-25'
GROUP BY channel;

7. Microsoft Fabric

Microsoft Fabric is a SaaS analytics platform that unifies data engineering, warehousing, real-time analytics and Power BI on one copy of data in OneLake.

Fabric is bought as capacity (F SKUs) shared by all workloads in assigned workspaces. Typical choice today: Azure Databricks for complex engineering and ML, Fabric for Microsoft/Power BI-centric analytics. Many organisations combine them, since both speak Delta.

8. Streaming: Event Hubs, Stream Analytics & Event Grid

# pip install azure-eventhub azure-identity
import json
from azure.eventhub import EventHubProducerClient, EventData
from azure.identity import DefaultAzureCredential

producer = EventHubProducerClient(
    fully_qualified_namespace="evh-acme-dev.servicebus.windows.net",
    eventhub_name="txn-events",
    credential=DefaultAzureCredential())          # managed identity / az login, no keys

with producer:
    batch = producer.create_batch(partition_key="ACC000001")   # same key → same partition → ordered
    batch.add(EventData(json.dumps({"txn_id": "T1", "account_id": "ACC000001", "amount": 2500})))
    producer.send_batch(batch)

# Spark reading Event Hubs through its Kafka endpoint
EH_NS = "evh-acme-dev.servicebus.windows.net"
conn = dbutils.secrets.get("kv-acme-dev", "evh-listen-connection-string")
jaas = ('kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required '   # Databricks
        f'username="$ConnectionString" password="{conn}";')                              # (drop "kafkashaded." elsewhere)

stream = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", f"{EH_NS}:9093")
    .option("subscribe", "txn-events")                   # Event Hub name = Kafka topic
    .option("kafka.security.protocol", "SASL_SSL")
    .option("kafka.sasl.mechanism", "PLAIN")
    .option("kafka.sasl.jaas.config", jaas)
    .option("startingOffsets", "latest")
    .load())

-- Stream Analytics: per-minute totals by channel, using event time
SELECT
    System.Timestamp() AS window_end,
    channel,
    COUNT(*)           AS txns,
    SUM(amount)        AS total_amount
INTO [adls-gold-output]
FROM [evh-txn-input] TIMESTAMP BY event_time
GROUP BY channel, TumblingWindow(minute, 1)

9. Databases: Azure SQL & Cosmos DB

Cosmos DB essentials

• Partition key is the most important design choice: high cardinality, even read/write spread, and present in most queries (e.g. /accountId). A logical partition is limited to 20 GB.
• Request Units (RU/s) measure throughput; provision them (manual or autoscale) or use serverless. A point read of a 1 KB item costs about 1 RU; cross-partition queries cost much more.
• Five consistency levels, from strongest to weakest: Strong, Bounded staleness, Session (the default), Consistent prefix, Eventual.
• Change feed streams inserts and updates in order per partition: use it to sync to the lake, trigger Functions, or build materialised views. Mirroring can replicate Cosmos DB into Fabric OneLake for analytics.

10. Security & Governance: Key Vault, Networking, Purview

az keyvault create -n kv-acme-dev -g rg-lakehouse-dev -l centralindia --enable-rbac-authorization true
az keyvault secret set --vault-name kv-acme-dev -n sql-etl-password --value "<from a secure source>"
az role assignment create --assignee-object-id "$ADF_MI" --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "$(az keyvault show -n kv-acme-dev --query id -o tsv)"

• Key Vault stores secrets, keys and certificates. ADF linked services, Databricks secret scopes and Functions read secrets at run time, so nothing sensitive sits in code. Customer-managed keys for storage encryption also live here.
• Private endpoints give a PaaS service (storage, Key Vault, SQL, Event Hubs) a private IP in your VNet; then disable public network access. Use NSGs and Azure Firewall to control traffic, and Private DNS zones so names resolve to private IPs.
• Microsoft Purview scans sources into a data map, classifies sensitive data (PAN, Aadhaar, emails), shows lineage from ADF, Synapse and Databricks, and provides a business glossary and catalog. In Databricks, Unity Catalog governs tables, with Purview giving the estate-wide view.
• Microsoft Defender for Cloud scores security posture and detects threats; Azure Policy enforces rules such as approved regions and no public storage.

11. Monitoring & Cost

Send diagnostic settings from ADF, Databricks, Synapse, storage and Event Hubs to a Log Analytics workspace, then query and alert with KQL:

// failed ADF activities in the last 24 hours, by pipeline
ADFActivityRun
| where TimeGenerated > ago(24h) and Status == "Failed"
| summarize failures = count(), last_error = any(ErrorMessage) by PipelineName, ActivityName
| order by failures desc

// pipeline duration trend
ADFPipelineRun
| where Status == "Succeeded"
| extend minutes = datetime_diff('minute', End, Start)
| summarize avg(minutes), max(minutes) by PipelineName, bin(TimeGenerated, 1d)

• Azure Monitor alerts on metrics (Event Hubs throttled requests, ADF failed runs) or log queries notify via action groups (email, Teams, webhook, ITSM).
• Cost Management: budgets with alerts per subscription or resource group, cost analysis by tag, Azure Advisor recommendations.
• Save money with reservations or savings plans for steady compute, auto-pause (Synapse, serverless SQL, Databricks auto-termination), spot VMs for Databricks workers, lifecycle tiering in storage, and right-sized Event Hubs throughput units.

12. Infrastructure as Code & CI/CD

// lake.bicep: ADLS Gen2 account with secure defaults
param location string = resourceGroup().location
param env string

resource lake 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: 'stacmelake${env}'
  location: location
  sku: { name: 'Standard_ZRS' }
  kind: 'StorageV2'
  properties: {
    isHnsEnabled: true                // ADLS Gen2
    minimumTlsVersion: 'TLS1_2'
    allowBlobPublicAccess: false
    supportsHttpsTrafficOnly: true
  }
  tags: { env: env, owner: 'data-platform' }
}

output lakeId string = lake.id

az deployment group what-if -g rg-lakehouse-dev -f lake.bicep -p env=dev    # preview changes
az deployment group create  -g rg-lakehouse-dev -f lake.bicep -p env=dev

# azure-pipelines.yml: test, then deploy infrastructure
trigger:
  branches: { include: [main] }

pool: { vmImage: ubuntu-latest }

stages:
  - stage: Test
    jobs:
      - job: unit
        steps:
          - task: UsePythonVersion@0
            inputs: { versionSpec: '3.11' }
          - script: pip install -r requirements-dev.txt && pytest -q
  - stage: DeployDev
    dependsOn: Test
    jobs:
      - deployment: infra
        environment: dev                 # approvals and checks are configured on environments
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self
                - task: AzureCLI@2
                  inputs:
                    azureSubscription: sc-data-dev      # service connection (workload identity federation)
                    scriptType: bash
                    scriptLocation: inlineScript
                    inlineScript: az deployment group create -g rg-lakehouse-dev -f infra/lake.bicep -p env=dev

Terraform (azurerm and databricks providers) is equally common, especially in multi-cloud teams. Either way, keep infrastructure in Git, preview with what-if / plan in pull requests, and deploy through pipelines using service connections with federated credentials rather than stored secrets.

13. Reference Architecture: Azure Lakehouse

 SOURCES                 INGEST                    LAKE (ADLS Gen2, Delta)              SERVE
 Core banking (SQL) ──►  ADF (self-hosted IR,  ──► raw ─► bronze ─► silver ─► gold ──► Synapse serverless / SQL warehouse
 SaaS / APIs        ──►  incremental copy)          (Azure Databricks: Auto Loader,       Power BI (Direct Lake / Fabric)
 Files (SFTP)       ──►                              MERGE, SCD2, DQ checks)              Cosmos DB (low-latency lookups)
 Apps / devices     ──►  Event Hubs ─► Databricks Structured Streaming / Stream Analytics ──► alerts, real-time dashboards

 CROSS-CUTTING: Entra ID + managed identities · Key Vault · Unity Catalog + Purview · Private endpoints + VNet injection
                Azure Monitor / Log Analytics · Cost Management · Bicep/Terraform + Azure DevOps or GitHub Actions

• Batch: ADF copies changed rows (watermark or CDC) to raw/ by date; a Databricks job ingests to bronze with Auto Loader, cleans and MERGEs into silver, and builds gold marts. ADF or Databricks Jobs orchestrate the steps.
• Streaming: producers publish to Event Hubs (partitioned by account); Databricks reads via the Kafka endpoint into bronze and scores in silver; Capture keeps a raw copy in ADLS.
• Security: no public endpoints, every service uses a managed identity, secrets live in Key Vault, PII is masked with Unity Catalog column masks, and resources stay in Indian regions via Azure Policy.

14. AWS ↔ Azure ↔ GCP Service Map

15. Common Interview Questions