AWS File Transfer Services

Transfer Family SFTP/FTPS/AS2 for partners, processing files on arrival, DataSync for bulk migration, Storage Gateway, offline and accelerated transfer, security and audit.

1. Choosing a Transfer Service

2. AWS Transfer Family (SFTP / FTPS / AS2)

Transfer Family runs managed SFTP, FTPS, FTP and AS2 endpoints that read and write directly to S3 or EFS. Partners keep using their existing tools, while the files land straight in your lake.

aws transfer create-server --protocols SFTP --domain S3 \
  --identity-provider-type SERVICE_MANAGED \
  --endpoint-type VPC \
  --endpoint-details '{"VpcId": "vpc-0abc", "SubnetIds": ["subnet-0a", "subnet-0b"], "AddressAllocationIds": ["eipalloc-0x", "eipalloc-0y"]}' \
  --logging-role arn:aws:iam::111122223333:role/transfer-logging \
  --security-policy-name TransferSecurityPolicy-2024-01

aws transfer create-user --server-id s-1234567890abcdef0 --user-name partner_hdfc \
  --role arn:aws:iam::111122223333:role/transfer-partner-hdfc \
  --home-directory-type LOGICAL \
  --home-directory-mappings '[{"Entry": "/", "Target": "/acme-partner-inbound/hdfc"}]' \
  --ssh-public-key-body "ssh-ed25519 AAAAC3Nza... partner@hdfc"

# what the partner runs
sftp -i ~/.ssh/acme_key partner_hdfc@sftp.acme.com
sftp> put settlement_20261026.csv.pgp
# the file appears at s3://acme-partner-inbound/hdfc/settlement_20261026.csv.pgp

3. Processing Files on Arrival

 partner ──SFTP──► Transfer Family ──► s3://acme-partner-inbound/hdfc/  (inbound bucket, KMS)
                                            │  S3 "Object Created" → EventBridge rule
                                            ▼
                          Step Functions: decrypt (PGP) → validate (size, name, control totals)
                                         → copy to raw/ zone → Glue job → notify partner + ops
                                         → on failure: move to quarantine/, alert

• Agree a file contract with each partner: naming convention, format, encoding, a control/trailer record with row count and totals, and a completion marker for multi-file drops.
• Never process half-uploaded files: trigger on the marker file, or use workflows that run after the upload completes.
• Keep the original file unchanged in a raw/archive zone for audit, and make processing idempotent by file name or checksum.

4. AWS DataSync

DataSync moves large amounts of data online between on-premises storage (NFS, SMB, HDFS, object storage), other clouds (e.g. Azure Blob, Google Cloud Storage) and AWS storage (S3, EFS, FSx). It handles parallelism, retries, integrity verification, encryption and scheduling.

# on-premises NFS export → S3 (an agent runs near the source)
aws datasync create-location-nfs --server-hostname nas01.corp.local --subdirectory /exports/sas_data \
  --on-prem-config AgentArns=arn:aws:datasync:ap-south-1:111122223333:agent/agent-0abc

aws datasync create-location-s3 --s3-bucket-arn arn:aws:s3:::acme-lake-prod --subdirectory /raw/sas_migration \
  --s3-config BucketAccessRoleArn=arn:aws:iam::111122223333:role/datasync-s3

aws datasync create-task --source-location-arn <nfs-location-arn> --destination-location-arn <s3-location-arn> \
  --name sas-data-migration \
  --options VerifyMode=ONLY_FILES_TRANSFERRED,OverwriteMode=ALWAYS,TransferMode=CHANGED,BytesPerSecond=104857600 \
  --includes FilterType=SIMPLE_PATTERN,Value="*.sas7bdat|*.csv" \
  --schedule ScheduleExpression="cron(0 22 * * ? *)"

aws datasync start-task-execution --task-arn <task-arn>

TransferMode=CHANGED copies only new or modified files on each run, so a migration can run repeatedly until cutover. BytesPerSecond limits bandwidth so business traffic is not affected.

5. Storage Gateway, Offline Transfer & Fast Uploads

Rough transfer time: 100 TB over a dedicated 1 Gbps link at about 80% utilisation takes roughly 12 days, which is why very large one-off migrations sometimes use offline devices or higher-bandwidth links.

6. Security & Audit

• Encrypt in transit (SFTP/FTPS/TLS) and at rest (SSE-KMS on target buckets); use PGP for file-level encryption when partners require it.
• One IAM role (or session policy) per partner, scoped to their own prefix; separate inbound and outbound buckets.
• Restrict partner source IPs with security groups; rotate SSH keys; log every session to CloudWatch.
• Keep checksums or ETags and control totals to prove files arrived complete and unchanged.

7. Common Interview Questions