Cloud Storage Service
Building a cloud storage service from scratch makes explicit the design decisions behind distributed object storage: atomic write guarantees, efficient large file handling, access control that scales without per-request database lookups, and metadata indexing that handles prefix queries. This project implements core S3-equivalent functionality to internalize these problems.
Built a cloud object storage service from scratch in Python
- ›S3-compatible API: bucket management, object CRUD, multipart upload
- ›Presigned URL access control with HMAC signatures — no per-request database lookups
- ›Pluggable storage backend: local filesystem or configurable cloud backend
An S3-inspired distributed object storage service implementing bucket management, multipart upload, and HMAC-signed presigned URL access control. Built to understand implementation-level design decisions behind distributed object storage.
Understanding distributed object storage at the implementation level — not just the API level. S3 is used everywhere but its internal design (consistency model, multipart upload, presigned URLs) is opaque to API consumers. This project builds equivalent functionality from scratch to make those decisions explicit.
- Write operations must be atomic — partial writes must not be visible to readers
- Large file uploads must use multipart upload to avoid single-request size limits
- Access control must work without per-request database lookups (presigned tokens)
- Metadata indexing must support prefix-based listing without full scan
FastAPI service with four layers: HTTP API (S3-compatible endpoints), metadata layer (PostgreSQL for bucket/object metadata and access policies), object layer (configurable storage backend), and access control layer (HMAC-signed presigned URLs with TTL, verified without database lookup). Multipart: initiate → upload parts → complete (atomic assembly).