Object Storage - Everything You Need to Know: Part - 2
This detailed overview explains the challenges and solutions when handling large files using object storage, focusing on multipart uploads, pre-signed URLs, network limitations, and download optimizations. It also covers best practices for efficient and scalable upload/download workflows with considerations on cost, integrity, and user experience.
Uploading large files (e.g., videos, CAD files, drone footage) presents multiple limitations when using a single PUT request in object storage APIs:
- Hard upload limit: Single PUT requests cannot exceed 5GB due to the object storage API constraints.
- Atomicity and reliability issues: Uploads are "all or nothing." Network interruptions cancel the upload entirely, requiring a restart from zero—problematic over unreliable consumer networks.
- Throughput limitations: A single TCP connection's throughput is limited by congestion control (window size / round-trip time) and is sensitive to packet loss, capping speeds (e.g., to 30 Mbps despite higher network speeds).
- Poor user experience: Single PUT uploads provide rough progress tracking, no pause/resume capability, and fail completely on interruptions.
These challenges necessitate a multipart upload approach for large files.
Multipart upload divides the file into chunks (parts) uploaded independently and in parallel, improving resilience, speed, and user experience. The multipart upload process involves three main steps:
- The upload ID works as a staging area, so partial uploads are invisible until committed.
- Each part is sent via separate HTTP PUT requests, enabling parallelism and independent retries.
- The final
Complete call records the parts' metadata order without physically concatenating data, ensuring efficiency.
- There is a hard limit of 10,000 parts per multipart upload.
Choosing the part size requires balancing:
- The maximum file size supported = part size × 10,000 parts.
- Retry cost: larger parts mean costly retries on failure.
- Memory usage: the browser loads parts lazily (by slice method), so memory scales with part size × concurrency.
Recommended practice:
- Use a dynamic part size computed as
\max(\text{default_part_size}, \frac{\text{file_size}}{10,000})
$$.
- Typical defaults are **16 MB to 32 MB** per part, balancing speed and retry cost.
- Avoid very large parts (e.g., 500 MB) due to expensive retries on unreliable networks.
### ETag and Data Integrity
- For single PUT uploads, the ETag is usually the object's MD5 hash.
- For multipart uploads, the ETag is a composite hash, not a direct MD5 of the whole file; it includes the number of parts marked with a suffix (e.g., "-42").
- **Do not rely on multipart ETags for end-to-end data integrity**.
**Integrity solutions:**
- Use checksum features in modern SDKs (e.g., CRC32C, SHA-256 checksums per part).
- Alternatively, compute your own SHA-256 of the entire file on the client and store it as metadata for verification after upload.
## 3. Managing Abandoned Multipart Uploads
Incomplete multipart uploads accumulate storage costs invisibly:
- If uploads are abandoned midway (e.g., user closes the tab), parts remain in the staging area.
- These parts do not appear in bucket listings but incur monthly storage costs.
- This can result in significant unintended expenses over time.
**Mitigation:**
- Configure **lifecycle rules** on buckets to **abort/cancel incomplete multipart uploads after a set period**, such as **7 days**.
- This is critical to prevent ongoing storage billing for orphaned multipart uploads.
## 4. End-to-End Upload Architecture with Browsers and Backends
A scalable large file upload system leveraging multipart uploads and pre-signed URLs typically follows this flow:
| Step | Description |
|-------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|
| 1. Initialization API Call | Browser sends file metadata (name, size, type) to backend endpoint (e.g., POST /uploads/init). |
| 2. Backend Processing | Backend authenticates, calculates ideal part size, creates multipart upload on storage (gets upload ID), stores a "pending" record in database, and generates pre-signed URLs for each part (in batches). |
| 3. Browser Upload | Browser uploads all parts in parallel (typically 4-6 concurrent PUT requests) using pre-signed URLs directly to object storage bucket. Memory usage is efficient due to lazy file slicing. |
| 4. Completion API Call | After all parts upload, browser calls backend (e.g., POST /uploads/complete) with upload ID and list of parts with ETags for final commit. Backend calls `CompleteMultipartUpload` on storage and verifies uploaded object with a HEAD call. Updates status to "completed" and emits events (e.g., via queue or websocket) to notify clients. |
**Additional Notes:**
- Pre-sign URLs in manageable batches rather than all at once to avoid oversized API responses.
- Ensure **CORS configuration exposes the ETag header**, so JavaScript can access it to complete the upload.
- Backend supports resuming uploads by tracking incomplete uploads and querying uploaded parts.
## 5. Downloading Objects: Best Practices and Patterns
Downloading often dominates bandwidth usage, so optimization is crucial:
### Key Recommendations
- **Do not proxy/relay downloads through your backend server.** This ties up resources, increases latency, costs on bandwidth, and poor scalability.
- Instead, serve downloads **directly from object storage buckets or CDNs** with appropriate access controls.
### Download Patterns
| Pattern | Description | Use Case |
|---------------------|----------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|
| Public Bucket / CDN | Bucket is public or behind CDN; files identified by content-addressed keys (hash-based names). Long cache control headers enable effective caching. | Public assets like website images, scripts, or products. |
| Pre-Signed GET URLs | Backend generates time-limited pre-signed GET URLs for authorized access. | Private or protected content requiring authorization before download. |
#### Challenge with Pre-Signed URLs and CDN Caching
- Pre-signed URLs contain unique timestamps/signatures per request, thus differing URLs for the same file cause **very low CDN cache hit rates**.
- This leads to unnecessary egress costs and degraded performance.
**Solutions:**
- **Extend pre-signed URL expiration windows** to coalesce requests under the same URL (e.g., sign URLs valid for one hour).
- The more proper approach: **use CDN-native signed URLs or signed cookies**, where authorization happens at CDN edge nodes, enabling full caching and efficient authorization.
### Range Requests for Partial and Resumable Downloads
Object storage supports HTTP range requests with status **206 Partial Content**, allowing clients to request byte ranges instead of whole files. This underpins features like:
- **Resumable downloads:** Resume from last byte after interruptions.
- **Video scrubbing:** Seek video by fetching only the required segments.
- **Parallel segmented downloads:** Different connections fetch different byte ranges simultaneously to boost throughput (basis for download managers).
Range request support is natively built into object storage due to adherence to HTTP standards.
### Adaptive Video Streaming Architecture
Modern streaming uses:
- Transcoding source videos into multiple quality levels.
- Segmenting videos into short chunks (seconds long).
- A manifest file listing available chunks and qualities.
- Client players fetch manifest, then download chunks via range requests adapting quality dynamically based on network conditions (Adaptive Bitrate Streaming).
This architecture uses standard object storage plus CDN infrastructure for storage and delivery. The complex engineering lies in transcoding pipelines and streaming logic, not in the storage layer.
## 6. Cost Considerations for Object Storage
Cost factors influence architecture and usage patterns:
| Cost Component | Description | Relative Cost Impact |
|-----------------------|-------------------------------------------------------------|----------------------------------------------|
| Storage | Charge per GB/month for stored data | Baseline recurring cost |
| Operations | Per-request charges (PUT, GET, DELETE) | Writes (PUTs) ~10× more expensive than reads |
| Egress (Bandwidth out) | Data leaving the cloud provider (downloads/streaming) | Can be very expensive; impacts overall cost |
Notably, some providers like Cloudflare R2 offer **free egress**, which can dramatically reduce costs at scale.
## 7. Summary and Best Practices Recap
- Object storage avoids local file system limits by abandoning mutability features (no in-place edits, rename, folders).
- Objects consist of key, bytes, metadata under flat namespace (buckets).
- Use **high-entropy, meaningful keys encoding ownership/metadata** upfront.
- Metadata is immutable without rewriting the full object.
- Storage durability uses **erasure coding**, offering strong data protection.
- Backups/versioning are **not default**—need explicit management.
- Avoid expensive and slow bucket list operations by tracking file metadata in a separate database.
- For **small files**, server-streamed direct upload with size limits suffices.
- For **large files**, use **multipart uploads with pre-signed URLs** and dynamic part sizing.
- Always use **multipart lifecycle rules to abort abandoned uploads** automatically.
- Do not trust multipart upload ETags for data integrity; rely on checksums.
- For downloads, **avoid backend relaying**, use **pre-signed GET URLs with CDN caching**, enabling range requests for resumable and progressive downloads.
- For video streaming, leverage transcoding, chunking, manifests, and adaptive bitrate protocols combined with object storage and CDN.
- Monitor and optimize costs by balancing storage, operations, and egress charges.
---
This comprehensive approach ensures scalable, efficient, cost-effective, and user-friendly handling of large file uploads and downloads in modern web applications leveraging object storage systems.