Home/Projects/Cloud Storage Service
Prototype
Distributed SystemsInfrastructureCloud StorageSystems Programming

Cloud Storage Service

Building a cloud storage service from scratch makes explicit the design decisions behind distributed object storage: atomic write guarantees, efficient large file handling, access control that scales without per-request database lookups, and metadata indexing that handles prefix queries. This project implements core S3-equivalent functionality to internalize these problems.

recruiter perspective

Built a cloud object storage service from scratch in Python

  • ›S3-compatible API: bucket management, object CRUD, multipart upload
  • ›Presigned URL access control with HMAC signatures — no per-request database lookups
  • ›Pluggable storage backend: local filesystem or configurable cloud backend
View repo
Hiring Summary

An S3-inspired distributed object storage service implementing bucket management, multipart upload, and HMAC-signed presigned URL access control. Built to understand implementation-level design decisions behind distributed object storage.

Understanding distributed object storage at the implementation level — not just the API level. S3 is used everywhere but its internal design (consistency model, multipart upload, presigned URLs) is opaque to API consumers. This project builds equivalent functionality from scratch to make those decisions explicit.

Challenges Addressed
  • Write operations must be atomic — partial writes must not be visible to readers
  • Large file uploads must use multipart upload to avoid single-request size limits
  • Access control must work without per-request database lookups (presigned tokens)
  • Metadata indexing must support prefix-based listing without full scan
◆Cloud Storage ServiceDISTRIBUTED SYSTEMS/visual-placeholder

FastAPI service with four layers: HTTP API (S3-compatible endpoints), metadata layer (PostgreSQL for bucket/object metadata and access policies), object layer (configurable storage backend), and access control layer (HMAC-signed presigned URLs with TTL, verified without database lookup). Multipart: initiate → upload parts → complete (atomic assembly).

Stack
PythonFastAPIPostgreSQLRedisDocker
Decision
Presigned URLs with HMAC signatures over per-request auth checks
Per-request auth requires a database lookup on every GET/PUT. HMAC-signed tokens encode the permission, expiry, and resource path — verification is a constant-time signature check with no database round-trip.
Decision
PostgreSQL for metadata over a separate key-value store
Object metadata queries (list by prefix, filter by size, sort by upload time) require relational operations. PostgreSQL's B-tree indexes support prefix queries efficiently.
Tradeoff
Storage backend flexibility vs. implementation complexity
Chose: Pluggable backend with local filesystem default
An abstracted storage interface lets the service run locally (filesystem) or against real object storage (S3, GCS) without changing the API layer. Testable without cloud dependencies.
01Replication across storage nodes for durability guarantees
02Versioning support for object overwrites with rollback
03Lifecycle policies for automated object expiry