Amazon S3 and object storage explained

Object storage vs block and file storage

Three storage paradigms dominate cloud infrastructure, and picking the wrong one wastes money or breaks performance:

• Block storage (EBS volumes, local SSDs) — fixed-size chunks a server mounts as a disk. Low latency, random read/write. Best for databases and OS root volumes.
• File storage (EFS, NFS) — hierarchical paths shared across servers. Good for legacy apps that expect POSIX semantics.
• Object storage (S3, R2) — flat namespace of immutable blobs with metadata. Optimized for throughput and durability at scale, not millisecond random I/O. You read or write whole objects via HTTP APIs.

S3 advertises eleven nines of durability (99.999999999%) by replicating objects across multiple Availability Zones in a region. That does not mean instant availability during regional outages — pair cross-region replication or versioning with a real disaster recovery plan.

Core vocabulary

• Bucket — globally unique name (DNS label) scoped to one AWS account and region.
• Key — the object's path-like identifier inside the bucket; slashes are convention, not true directories.
• Object — data bytes plus system metadata (size, ETag, storage class) and optional user metadata headers.
• Region — where bytes physically live; egress to other regions costs money.
• Prefix — the substring before the last slash; used in IAM conditions and lifecycle filters (logs/2026/06/).

Storage classes and cost trade-offs

Not every object needs instant retrieval. S3 offers tiered classes; the wrong tier either overpays for cold archives or charges retrieval fees you did not budget:

• S3 Standard — default for frequently accessed data. Low latency, no retrieval fee.
• S3 Intelligent-Tiering — auto-moves objects between frequent and infrequent tiers based on access patterns; small monitoring fee per object.
• S3 Standard-IA / One Zone-IA — cheaper storage, retrieval surcharge; One Zone-IA skips cross-AZ replication (higher risk).
• S3 Glacier Instant / Flexible / Deep Archive — minutes-to-hours retrieval for backups and compliance archives; lowest $/GB.

Request pricing matters at scale: LIST, GET, PUT, and DELETE each bill per thousand calls. A bug that polls a bucket every second can cost more than the storage itself. Use lifecycle rules to transition prefixes to cheaper classes after N days and expire scratch data automatically.

Versioning and delete markers

Enable versioning on buckets holding irreplaceable data. A DELETE without a version ID inserts a delete marker instead of erasing history — accidental overwrites become recoverable. Pair versioning with lifecycle rules that expire noncurrent versions after a retention window so storage does not grow forever.

Access control: IAM policies, bucket policies, and ACLs

Public S3 bucket leaks remain a top cloud breach pattern. Modern setups block all public access at the account level and grant access only through explicit policies:

• IAM identity policies — attach to users, roles, or groups; grant s3:GetObject on arn:aws:s3:::my-bucket/invoices/*.
• Bucket policies — resource-based JSON on the bucket; common for cross-account access or CloudFront origin restrictions.
• ACLs — legacy per-object grants; avoid for new work unless a vendor requires them.
• Block Public Access — account- and bucket-level switches that override permissive ACLs or policies.

Principle of least privilege: scope actions to the prefix your service needs. A log-shipper role needs PutObject on logs/*, not s3:* on *. Use IAM condition keys for TLS-only (aws:SecureTransport) and source VPC endpoints where applicable.

Pre-signed URLs

When browsers or mobile clients must upload or download without holding AWS credentials, your API generates a pre-signed URL — a time-limited signed GET or PUT link. Set short expiry (minutes), restrict content-type and max size server-side before signing, and never log the full URL with credentials embedded. For downloads of private media, pre-signed GET beats making the bucket public.

Upload patterns: single PUT, multipart, and consistency

Objects up to 5 GB can use a single PutObject. Larger files should use multipart upload: split into parts (5 MiB minimum except the last), upload parts in parallel, then CompleteMultipartUpload. Failed uploads leave orphan parts that bill silently — lifecycle rules can abort incomplete multipart uploads after seven days.

S3 provides read-after-write consistency for new object PUTs in all regions: a successful upload is immediately visible to GET and LIST. Overwrites and deletes are eventually consistent for LIST in some edge cases — do not build distributed locks on LIST alone; use conditional writes with ETags or an external lock store like distributed locking.

For high-throughput ingest (video, telemetry), combine multipart upload with transfer acceleration or upload directly from the client via pre-signed URLs so bytes never pass through your API servers.

Events, static hosting, and the S3 API ecosystem

S3 can trigger downstream work without polling:

• Event notifications — on ObjectCreated or ObjectRemoved, fan out to SNS, SQS, Lambda, or EventBridge. A classic pattern: upload triggers a Lambda to virus-scan, thumbnail, or index metadata.
• Static website hosting — serve HTML/CSS/JS from a bucket behind CloudFront; pair with CDN caching for global latency.
• Inventory and analytics — daily CSV inventories of object sizes help audit storage growth; S3 Storage Lens aggregates cross-account metrics.

The S3 API is a de facto standard. MinIO runs on-premises; Cloudflare R2 offers zero egress to Workers; tools like aws s3 cp, boto3, and rclone work against any compatible endpoint by changing the base URL. That portability reduces vendor lock-in — design keys and lifecycle policies to migrate between providers.

Worked example: Harbor Supply invoice archive

Harbor Supply generates 12,000 PDF invoices per month. Requirements: seven-year retention, finance team download by invoice ID, no public exposure, and nightly analytics on aggregate file counts.

1. Bucket layout — harbor-invoices-prod in us-east-1; keys year=2026/month=06/inv-{id}.pdf (Hive-style prefixes help Athena queries later).
2. Upload path — billing service assumes IAM role with s3:PutObject on year=*/month=*/*; server-side encryption AES-256 (SSE-S3) enabled by default bucket policy.
3. Download path — finance portal calls internal API; API validates user role, then returns a 15-minute pre-signed GET for the exact key.
4. Lifecycle — after 90 days, transition to Standard-IA; after 365 days, Glacier Flexible Retrieval; expire noncurrent versions after 30 days.
5. Events — ObjectCreated to SQS; worker updates search index and increments Prometheus counter for ops dashboards.
6. Guardrails — Block Public Access on; deny insecure transport; AWS CloudTrail data events on the bucket for audit.

Monthly storage cost lands under $40 for ~500 GB with IA/Glacier tiering — far cheaper than keeping PDFs on EBS volumes attached to app servers, and far more durable than a single NFS mount.

Storage decision table

Common pitfalls

• Public bucket misconfiguration — Principal: "*" on s3:GetObject exposes every object; enable Block Public Access and audit with tools like AWS Config.
• Listing huge buckets — ListObjectsV2 without a prefix paginates millions of keys and burns request budget; always filter by prefix.
• Hot partition keys — writing all objects under one prefix (uploads/latest/) can throttle throughput; distribute with hash prefixes.
• Ignoring egress — serving 10 TB/month to the public internet from S3 directly is expensive; put CloudFront or another CDN in front.
• Orphan multipart parts — failed uploads accumulate; abort via lifecycle or scheduled cleanup.
• Secrets in object metadata — user metadata is not encrypted separately and may appear in logs; store secrets in Secrets Manager, not S3 headers.
• Cross-region surprise bills — replication and inter-region COPY charge transfer fees; pick one primary region deliberately.

Production checklist

• Enable Block Public Access at account and bucket level; verify with external scanner.
• Default encryption (SSE-S3 or SSE-KMS); consider KMS keys per environment.
• Versioning on for irreplaceable data; lifecycle to expire noncurrent versions.
• IAM least privilege by prefix; no wildcard s3:* on production roles.
• CloudTrail data events or equivalent audit log for sensitive buckets.
• Lifecycle transitions and abort-incomplete-multipart rules documented in IaC.
• Pre-signed URL expiry under 1 hour; validate content-type and size before signing.
• Monitor bucket size, request rate, and 4xx/5xx via CloudWatch; alert on sudden spikes.
• Cross-region replication or backup only where RPO requires it — document cost.
• Pair with Terraform IaC so bucket policies are reviewed in PRs, not click-ops.

Key takeaways

• Object storage addresses blobs by bucket + key; it optimizes for durability and scale, not POSIX disk semantics.
• Match storage class to access pattern; lifecycle rules automate tiering and expiry.
• IAM policies and bucket policies — not public ACLs — should gate every read and write.
• Pre-signed URLs and multipart upload are the standard patterns for client uploads and large files.
• The S3 API is portable across AWS, R2, and MinIO — design prefixes and encryption for migration from day one.

Related reading

• ETL and ELT data pipelines explained — landing zones and warehouse ingest from object storage
• Disaster recovery and backup strategies explained — RPO, RTO, and cross-region failover
• Serverless computing explained — Lambda triggers on S3 events
• Data warehouses and lakehouses explained — query parquet files in place over S3