Object Storage - Everything You Need to Know: Part -1

This video transcript explores the challenges of handling user file uploads in production-grade backend systems, critiques naive approaches, introduces object storage as a solution, and details the best practices and architectural patterns for managing large files reliably and efficiently. It emphasizes key distributed system concepts, storage trade-offs, and innovative features like pre-signed URLs and conditional writes.

1. Problems with Naive File Upload Handling

The intuitive but flawed pattern for file uploads involves receiving a file via form data, saving it on the backend server’s local filesystem, and storing the file path in a database. While this works in simple cases, it breaks under real production conditions due to:

• Ephemeral server storage: Containers running backend instances have ephemeral filesystems that lose files on restarts or crashes.
• Memory constraints: Loading large files (e.g., multi-GB videos) directly into server memory causes out-of-memory (OOM) failures.
• Horizontal scaling inconsistency: With multiple backend instances behind a load balancer, files stored locally on one instance become inaccessible if requests route to a different instance, causing 404 errors.
• Fixed disk size and vertical scaling limits: Expanding disk storage vertically requires downtime and has upper size limits.
• Durability risks: Single disks can fail, losing all stored files without replication.
• Resource-intensive downloads: Serving large files from the backend consumes server CPU and IO resources inefficiently.
• Lack of transactional atomicity: Files and database entries are managed separately without atomic transactions, causing orphaned files or broken links during failures.

The root cause is treating files like typical JSON data within backend servers, which ignores files’ fundamentally different access patterns, durability needs, and scale challenges.

2. Overview of Storage Types and Trade-offs

Understanding object storage requires comparing it with other storage types:

Key Object Storage Characteristics

• Minimal interface reduces complexity and coordination overhead, enabling planet-scale (xabytes) capacity.
• Objects are immutable: no modifications or partial updates; whole-object replacements only.
• No real folders/directories; folder structure is an illusion created by key naming conventions with delimiters (e.g., /).
• Operations are HTTP requests crossing network boundaries, leading to higher latency than block storage.
• Storage systems like Amazon S3 provide strong read-after-write consistency guaranteeing immediate visibility of uploads after completion.

3. Object Storage Terminology and Structure

• An object has four components:

  • Key: Unique string identifier (full path-like string).
  • Value: Binary blob (the data).
  • System metadata: Size, content-type, last modified timestamp, eTag (a unique fingerprint).
  • User metadata: Arbitrary key-value pairs the user attaches for custom info (e.g., uploader ID).

• Objects live inside buckets, which are containers with globally unique names.

• Folder-like hierarchies are formed by key prefixes but are not actual directories.

• Empty folders do not natively exist; they can be mimicked by zero-byte objects with special keys.

4. Critical Object Storage Design Patterns

Key Design Rules

• Never use user-supplied file names as keys. Risks include name collisions, path traversal exploits, and unsupported special characters.
• Generate random unique keys (UUID or hashes) and store original names separately if needed.
• Put high-entropy (random) prefixes at the start of keys to avoid hot partitions caused by sequential keys (e.g., timestamps) overloading index partitions.
• Design keys to embed ownership info (e.g., tenant ID/user ID) to simplify access control via prefix matching.

Behind-the-Scenes Storage Architecture

• Two planes:
  • Data plane: Stores raw object bytes, uses replication or erasure coding for durability. Has simpler, large-volume operations.
  • Metadata plane: Maintains a distributed, strongly consistent index mapping keys to physical storage locations. This is a complex, small-volume/high-query-rate problem.

Durability via Erasure Coding

• Unlike simple replication that triples storage cost, erasure coding divides data into shards and parity shards enabling recovery from multiple failures with less overhead.
• Example: 14 data shards + 6 parity shards spread across machines in different failure domains.
• AWS S3 durability promise – 11 9s (99.999999999%) means loss of a single object is expected only once every 10,000 years for 10 million objects.

Strong Consistency and Conditional Writes

• Recent S3 updates enable:
  • Strong read-after-write consistency: Immediate availability after upload without retry delays.
  • Conditional writes: Headers like If-None-Match: * allow atomic creation only if object does not exist, preventing silent overwrites and enabling distributed locking semantics.
  • If-Match header supports compare-and-swap semantics for safe concurrent updates.

5. Upload Architectures

5.1. Backend-Proxy Upload (Naive)

• The browser uploads file to backend API.
• Backend buffers full file in-memory and then uploads to object storage.
• Suitable only for small files.
• Causes memory exhaustion at scale or with large files (streaming partially improves but still requires server to be a middleman).
• Backend resources are wasted on slow network transfers from client to server.

5.2. Pre-Signed URLs (Recommended Production Pattern)

• Backend issues a pre-signed URL: a URL with embedded HMAC-based authorization valid for limited time and specific operations.
• The client uploads directly to the object storage bucket via this URL.
• Backend never handles file bytes — eliminates memory pressure and scales flawlessly.
• Bucket remains private; credentials never leak to client.
• URLs are tightly scoped by method, key, expiry time, headers.

Controlling Upload Constraints

• Use pre-signed POST URLs with policy documents to enforce:
  • Content length size range (min/max).
  • Content type prefix (e.g., image/*).
  • Key prefix restrictions (to prevent folder escape).
• Object storage verifies these conditions and rejects violators before upload.

5.3. Handling Uploads with Database Sync

• Upload implemented as two-phase flow:
  1. Backend generates object key, inserts pending record in database, issues pre-signed URL to client.
  2. Client uploads directly to object storage.
  3. Client calls completion API to notify backend.
  4. Backend verifies object existence and properties via HEAD request to storage.
  5. If validated, backend sets database record to “completed.”
• Provides strong linkage between database state and object storage contents.
• Requires background cleanup jobs to prune orphaned pending uploads.

5.4. CORS Setup

• Since browser uploads cross origins, CORS policy must allow specific frontend origins, methods, and expose headers like eTag for multipart uploads.

6. Download Considerations

• Downloading large files from backend servers is inefficient and wastes compute resources.
• Better served by CDNs or direct access to object storage URLs to reduce backend load.
• Range requests enable partial downloads and efficient video seeking.

7. Summary of Key Insights

• Local filesystem storage is brittle, not scalable, and breaks containerized deployments.
• Object storage offers a scalable, durable, and consistent way to manage files at infinite scale.
• Immutable objects and minimal interface reduce complexity and remove distributed coordination issues impacting POSIX file systems.
• Proper key design avoids performance bottlenecks and security risks.
• Pre-signed URLs provide secure, direct client-to-cloud uploads bypassing backend memory and bandwidth limits.
• Distributed systems features like conditional writes enable building sophisticated workloads (distributed locks, write-ahead logs) on top of object storage.
• Strongly consistent metadata planes are a challenging but foundational part of cloud object storage.
• Effective synchronization between database metadata and physical object data requires a two-phase flow plus cleanup mechanisms.

Object storage represents an evolution from traditional file handling toward robust, scalable cloud-native file management and underpins many modern backend production systems handling large multimedia or user-generated content.

This architecture and set of patterns provide the production-grade blueprint for managing uploads/downloads securely, efficiently, and fault-tolerantly in cloud environments.